Next Article in Journal
Technologies and Principles of Hot Recycling and Investigation of Preheated Reclaimed Asphalt Pavement Batching Process in an Asphalt Mixing Plant
Next Article in Special Issue
Interpretation US Elastography in Chronic Hepatitis B with or without Anti-HBV Therapy
Previous Article in Journal
Settlement Prediction of Footings Using VS
 
 
Search for Articles:
Title / Keyword
Author / Affiliation / Email
Journal
Article Type
 
 
Advanced Search
 
Section
Special Issue
Volume
Issue
Number
Page
 
Journals
Applied Sciences
Volume 7
Issue 11
10.3390/app7111103
applsci-logo
Submit to this Journal Review for this Journal Propose a Special Issue
► Article Menu

Article Menu

share Share announcement Help format_quote Cite question_answer Discuss in SciProfiles thumb_up
...
Endorse textsms
...
Comment
first_page
settings
Order Article Reprints
Font Type:
Arial Georgia Verdana
Font Size:
Aa Aa Aa
Line Spacing:
Column Width:
Background:
Article

Shear Wave Elastography Combining with Conventional Grey Scale Ultrasound Improves the Diagnostic Accuracy in Differentiating Benign and Malignant Thyroid Nodules

by
Faisal N. Baig
1,
Shirley Y. W. Liu
2,
Hoi-Chun Lam
1,
Shea-Ping Yip
1,
Helen K. W. Law
1,* and
Michael Ying
1,*
1
Department of Health Technology and Informatics, The Hong Kong Polytechnic University, Hung Hom, Kowloon, Hong Kong, China
2
Department of Surgery, Prince of Wales Hospital, The Chinese University of Hong Kong, Shatin, New Territories, Hong Kong, China
*
Authors to whom correspondence should be addressed.
Appl. Sci. 2017, 7(11), 1103; https://doi.org/10.3390/app7111103
Submission received: 11 September 2017 / Revised: 30 September 2017 / Accepted: 13 October 2017 / Published: 25 October 2017
(This article belongs to the Special Issue Ultrasound Elastography)
Browse Figures
Versions Notes

Abstract

:
Shear wave elastography provides information about the stiffness of thyroid nodules that could be a new indicator of malignancy. The current study aimed to investigate the feasibility of using shear wave elastography (SWE) alone and in conjunction with grey scale ultrasound (GSU) to predict malignancy in 111 solitary thyroid nodules. Malignant thyroid nodules tended to have microcalcification, hypoechogenicity, tall to width ratio >1, and irregular borders (p < 0.05). SWE indices (Emaximum and Emean) of malignant nodules (median ± standard error: 85.2 ± 8.1 kPa and 26.6 ± 2.5 kPa) were significantly higher than those of benign nodules (median ± standard error: 50.3 ± 3.1 kPa and 20.2 ± 1 kPa) (p < 0.05). The optimal cut-off of Emaximum and Emean for distinguishing benign and malignant nodules was 67.3 kPa and 23.1 kPa, respectively. Diagnostic performances for GSU + Emaximum, GSU + Emean, GSU, Emaximum and Emean were: 70.4%, 74.1%, 96.3%, 70.4% and 74.1% for sensitivity, 83.3%, 79.8%, 46.4%, 70.2%, and 66.7% for specificity, and 80.2%, 78.4%, 58.5%, 70.3%, and 68.5% for accuracy, respectively. Our results suggested that combining GSU with SWE (using Emaximum or Emean) increased the overall diagnostic accuracy in distinguishing benign and malignant thyroid nodules.

1. Introduction

Grey scale ultrasound (GSU) is the first-line imaging investigation for the assessment of thyroid nodules [1,2]. Although certain GSU features have been reported to be associated with thyroid malignancy (microcalcification, hypoechogenicity, irregular shape, tall to width ratio >1, absent halo sign, and irregular margins) [3,4,5], these features may also be found in benign nodules [6,7]. To date, none of the GSU features are known to exhibit a high sensitivity and specificity in differentiating benign and malignant thyroid nodules [8,9,10].
Fine-needle aspiration cytology (FNAC) is recommended by the American Thyroid Association to evaluate thyroid nodules when the GSU findings are inconclusive [11]. However, potential sampling and analytical errors limit its application in establishing correct diagnosis in some cases such as small thyroid nodules [12,13]. Only about 65–75% of thyroid nodules can be correctly diagnosed by FNAC [14], whilst the remaining 25–35% of nodules yield indeterminate pathology or non-diagnostic results [15]. Both GSU and FNAC have wide ranges of sensitivity (52–97% and 54–90%, respectively), and specificity (26–83% and 60–98%, respectively) in distinguishing benign and malignant thyroid nodules [10,16].
Power Doppler and color Doppler ultrasound evaluate the vascularity of thyroid nodules. It is generally believed that hypervascularity is a feature of thyroid malignancy [17,18]. Some other studies suggested that vascular pattern of thyroid nodules can be useful to predict the thyroid malignancy e.g., thyroid nodules with dominant central vascularity are frequently associated with malignant thyroid nodules, whereas benign thyroid nodules tend to have peripheral vascularity or appear avascular [19,20]. Ultrasound elastography is a non-invasive imaging technique that measures tissue stiffness and provides color-coded elasticity map. In strain elastography (SE), the elasticity color map and strain ratio (SR) provide qualitative and semi-quantitative estimates of elasticity distribution, respectively. SE has limited clinical value due to the need of subjective interpretation of elastogram and compressive maneuvers. Contrarily, shear wave elastography (SWE) is less operator-dependent and more reproducible than SE, and it provides quantitative assessment in the form of elasticity indices (Emaximum, Emean, Eminimum). SWE uses focused ultrasound impulses at varying depths in tissue to induce tissue displacement and results in the generation of shear waves which propagate laterally [21]. Shear waves travel faster in harder medium than in softer counterpart, and their velocity is directly proportional to the square root of Young’s modulus. Young’s modulus measures the tissue stiffness by demonstrating the relationship between stress (applied force) and strain (resultant deformation) [22,23]. Stiffer tissues exhibit higher Young’s modulus as compared to softer tissues. In the current study, a higher Young’s modulus for malignant nodules (85.2 ± 8.1 kPa) than benign nodule (50.3 ± 3.1 kPa) is observed (p < 0.05). Assuming the tissue density 1 g/cm3, the Poisson’s ratio was 0.5 [24,25]. The propagation speed of shear waves is tracked by the ultrafast sonographic tracking technique [26]. By evaluating the speed of shear waves in the tissue, the tissue stiffness can be determined. In comparison to the normal thyroid parenchyma, most malignant thyroid nodules have firm stroma due to the presence of excessive collagen, excessive myofibroblast, and desmoplastic transformation, hence resisting tissue strain upon stress [10,27,28]. Therefore, most of the malignant thyroid nodules tend to be stiffer. As the stiffness of thyroid nodules can be evaluated with shear wave elastography, malignant thyroid nodules tend to have higher shear wave elastography index. In the current study, we hypothesize that malignant thyroid nodules are stiffer than benign nodules on SWE. The present study aimed to evaluate the diagnostic accuracy of SWE in differentiating benign and malignant thyroid nodules when it was used alone and combined with GSU. Positive study results will have implications in the identification of thyroid nodules that need to be further evaluated by FNAC/biopsy, and thus a correct diagnosis can be established.

2. Materials and Methods

2.1. Subject Recruitment

This prospective study was approved by the Human Subject Ethics Sub-committee (HSESC), the Hong Kong Polytechnic University, Hong Kong, and Institutional Review Board of Prince of Wales Hospital, Hong Kong. Informed written consent for the study was obtained from all of the human subjects in accordance with the WORLD Medical Association Declaration of Helsinki: Ethical principles for medical research involving human subjects, 2008 (http://www.wma.net/en/30publications/10policies/b3/). The privacy rights of the human subjects in the study were observed.
Between September 2013 and June 2015, a total of 122 consecutive patients (22 men and 100 women, mean age: 53 ± 13.7 years; age range 21–95 years) with 163 thyroid nodules were recruited at the Prince of Wales Hospital, Hong Kong. We included patients who had at least one thyroid nodule diagnosed clinically or radiologically on thyroid ultrasound examination. Exclusion criteria were completely cystic nodules or any forms of inflammatory thyroid diseases (acute thyroiditis, chronic thyroiditis, grave’s disease, and sub-acute thyroiditis), which were associated with increased thyroid parenchymal stiffness.

2.2. Ultrasound Examination of Thyroid Gland

GSU and SWE examinations of thyroid gland were performed on both sides of the neck of all patients. Due to the ethical issues and restricted policies of our institution, we had limited data access to the patient’s clinical reports. Therefore, we could not check the stage of tumors of the patients. However, we had accessed to the histopathology and fine needle aspiration cytology results for the final diagnosis of the nodules. Cytopathology and/or histopathology diagnoses were used to correlate the accuracy of GSU and SWE findings. All of the thyroid ultrasound examinations were conducted by the same operator (M.Y.) using the same ultrasound unit in conjunction with a 4–15 MHz linear transducer (Aixplorer, Supersonic Imagine, Aix-en-Provence, France). All of the ultrasound examinations were performed with an imaging mechanical index (MI) of 1.5 and the MI of the “push” pulses for the SWE was 1.2. In the thyroid ultrasound examinations, we used a broadband frequency ultrasound transducer, which allows operators to choose different frequencies for examination. In the current study, we standardized the scanning protocol of using high frequency (~15 MHz) to optimize the image quality. During the ultrasound examination, patients were asked to lie in supine position on the examination couch with the neck and shoulders supported by pillow to keep patient’s neck slightly hyperextended. A generous amount of coupling gel was applied and GSU was performed to identify any nodule in the thyroid lobes and isthmus. When a thyroid nodule was identified (based on perceived contrast between the echogenicity of thyroid nodule and adjacent thyroid parenchyma), multiple transverse, and longitudinal sonograms of the nodule were obtained. Each nodule was assessed for suspicious grey scale sonographic features [1,21,29,30] including absent halo sign, microcalcification, hypoechogenicity, internal solid echotexture, tall to width ratio >1, and irregular margins (Figure 1 and Figure 2).
After GSU, SWE was then performed on the thyroid nodule. For patients with multiple thyroid nodules, SWE was performed on the largest thyroid nodule, and/or the nodule with one or more suspicious grey scale sonographic features, as described above. In the SWE examination, the size of the SWE acquisition box was first adjusted to cover the entire thyroid nodule. Multiple transverse and longitudinal shear wave elastograms of the nodule were obtained (Figure 3 and Figure 4).
To acquire an elastogram, the transducer was held at the same position for at least 2 s to allow the SWE signal acquisition to settle and to reduce variability. During SWE examination, caution was made to avoid compression applying on the patient’s neck by the transducer, which might affect the stiffness measurement of thyroid nodules. Taking shear wave elastograms in the scan planes showing calcification within the nodule were avoided. After the ultrasound examination, FNAC was performed on the targeted nodules. Nodules with confirmed malignancy (papillary carcinoma, follicular carcinoma, or medullary carcinoma) or indeterminate cytology (follicular neoplasm, follicular lesion of undetermined significance, atypia of undetermined significance, or repeated non-diagnostic cytology) were further evaluated by surgical resection and histopathological examination.
Before the commencement of the main study, two independent observers (F.N.B. and H.C.L.) conducted an inter-observer reliability test on 50 archived data sets of randomly selected thyroid nodules to evaluate the inter-observer reliability in SWE measurement. The results showed that there was a high inter-observer reliability between the two observers (intraclass correlation coefficient = 0.98). In the current study, archived grey scale sonograms and elastograms were reviewed by single observer who was blinded to the cytology and histology results. Each thyroid nodule was assessed for the presence or absence of suspicious grey scale sonographic features: absent halo sign, microcalcification, hypoechogenicity, internal solid echotexture, tall to width ratio >1, and irregular margins. The hypoechogenicity of thyroid nodules was determined by observer’s visual assessment based on the perceived contrast between thyroid nodule and adjacent thyroid parenchyma.
The stiffness of the thyroid nodule was measured using the inbuilt quantification tool of the ultrasound unit (Q-boxTM, Supersonic Imagine, Aix-en-Provence, France) on the elastograms. The circular quantification region of interest (ROI) was used to cover the entire thyroid nodule without including the adjacent thyroid parenchyma. The software then automatically calculated the SWE indices (Emaximum, Emean, and Eminimum), which were expressed in kilo-pascals (kPa) (Figure 3 and Figure 4). The range of stiffness measurement ranged from 0 kPa to ≥100 kPa. For each thyroid nodule, five measurements with the highest stiffness values were selected from both longitudinal and transverse elastograms, and the average of them was calculated to deduce the mean of Emaximum, Emean and Eminimum.

2.3. Data Analysis

Chi square test was used to determine the significance of difference of GSU features between benign and malignant nodules, whereas the significance of difference of SWE indices between benign and malignant nodules was calculated by Mann Whitney U test. The diagnostic performance of GSU was evaluated by deducing the frequency tables of true-positive, true-negative, false-positive, and false-negative cases. Receiver operating characteristic (ROC) curves were used to determine the optimal cut-off of different SWE indices in distinguishing benign and malignant nodules, and the associated diagnostic performance of the optimal cut-off. The diagnostic performance of combining GSU and SWE was determined based on the principle that a thyroid nodule was considered malignant when it was presented with at least one suspicious GSU feature (i.e., hypoechogenicity, tall to width ratio ≥1, irregular border or microcalcification) and had SWE indices (Emaximum ≥ 67.3 kPa or Emean ≥ 23.1 kPa) equal to or greater than the corresponding optimal cut-off values. Thyroid nodules did not fulfill these criteria were categorized as benign. All statistical analyses were performed using the Statistical Package for the Social Sciences (SPSS) software (Version 20, IBM Corporation, Armonk, NY, USA) and two-tailed p value < 0.05 was significant.

3. Results

3.1. Histology Results

In the 122 thyroid nodules evaluated, 73 nodules were confirmed as benign on FNAC. In the remaining 49 nodules, histopathological examination upon surgical resection confirmed 27 nodules to be malignant and 11 benign. The remaining 11 nodules were excluded from the study because surgical resection had not been performed and the final diagnosis could not be obtained. Therefore, altogether, 111 thyroid nodules (27 malignant and 84 benign) were included in this study. Amongst the 27 malignant nodules, there were 23 papillary thyroid carcinomas, 3 follicular thyroid carcinomas and 1 Hurthle cell carcinoma.

3.2. Grey Scale Ultrasound

The grey scale sonographic features of thyroid nodules are summarized in Table 1. Among different grey scale sonographic features, microcalcification (77.8% and 7.1%, respectively), tall to width ratio >1 (59.3% and 13.1%, respectively), hypoechogenicity (92.6% and 33.3%, respectively) and irregular margin (55.6% and 16.7% respectively), were significantly more common in malignant nodules than benign nodules (all p < 0.05). There was no significant difference in the absent halo sign and internal solid echotexture between malignant and benign nodules (p > 0.05).
With the above results, further data analysis was performed to determine the diagnostic performance of GSU in distinguishing benign and malignant thyroid nodules. In the data analysis, thyroid nodules with at least one of the above four significant grey scale sonographic features (i.e., microcalcification, tall to width ratio >1, hypoechogenicity and irregular margin) were malignant, whereas others were benign. Using these assessment criteria, 26 malignant nodules and 39 benign nodes were correctly identified. Results showed that the sensitivity, specificity, positive predictive value (PPV), negative predictive value (NPV), and overall accuracy of GSU in distinguishing benign and malignant nodules were 96.3%, 46.4%, 36.6%, 97.5% and 58.5% respectively (Table 2).

3.3. Shear Wave Elastography

Table 3 shows the SWE indices of benign and malignant thyroid nodules. The results showed that malignant thyroid nodules were associated with higher SWE indices. The median of Emaximum of malignant nodules (85.2 ± 8.1 kPa) was significantly higher than that of benign nodules (50.3 ± 3.1 kPa) (p < 0.05). Similarly, the median of Emean of malignant nodules was 26.6 ± 2.5 kPa and of benign nodules was 20.2 ± 1 kPa, and the difference was statistically significant (p < 0.05). However, no significant difference was found in Eminimum between benign and malignant nodules (p > 0.05).
Since significant difference was found between benign and malignant nodules in Emaximum and Emean only, the evaluation of diagnostic accuracy was performed in these two SWE indices. With the use of ROC curves (Figure 5), the optimal cut-off of Emaximum and Emean in distinguishing benign and malignant nodules were 67.3 kPa and 23.1 kPa, respectively. Using the optimal cut-off of Emaximum, 19 malignant and 59 benign nodules were correctly evaluated. The sensitivity, specificity, and overall accuracy of Emaximum were 70.4%, 70.2%, and 70.3%, respectively. Using the optimal cut-off of Emean, 20 malignant and 56 benign nodules were correctly assessed, and the sensitivity, specificity, and overall accuracy of Emean were 74.1%, 66.7%, and 68.5%, respectively (Table 2).

3.4. Combination of Grey Scale Ultrasound and Shear Wave Elastography

Result showed that GSU had a high sensitivity (96.3%) but a low specificity (46.4%), leading to an overall accuracy of 58.5% in assessing thyroid nodules. When GSU combined with SWE (Emaximum or Emean), the overall accuracy increased to 80.2% for Emaximum and 78.4% for Emean with a sensitivity of 70.4% and 74.1% and specificity of 83.3% and 79.8%, respectively (Table 2, Figure 6). When GSU combined with Emaximum or Emean, 19 or 20 malignant and 70 or 67 benign thyroid nodules were correctly identified, respectively.

4. Discussion

Differential diagnosis of thyroid nodules to predict malignancy poses a diagnostic dilemma in clinical settings. FNAC and histopathology examination of surgical specimen of thyroid nodules are the standard procedures to diagnose thyroid cancer. FNAC has a false-positive rate of 1–11.5% and a false-negative rate about 7.7%, whilst 20–30% of cases are non-diagnostic [31,32]. However, the technique is highly dependent on the experience of the pathologists and the adequacy of the samples attained. Moreover, histopathological features of malignant thyroid nodules overlap with benign nodules, and thus there is a high incidence of “skip diagnosis”. GSU is commonly used to assess thyroid nodules but no single GSU feature can accurately predict thyroid malignancy [33]. SWE is a novel ultrasound technique that measures tissue elasticity by tracking shear wave propagation through body tissues and provides quantitative measurements. The technique is less operator-dependent and highly reproducible [10]. SWE evaluation of thyroid nodules has been documented. However, the reported cut-off levels to differentiate benign and malignant nodules were variable, which was probably due to different methodologies used in the studies. Among the available literature, the highest cut-off values of SWE indices for evaluating thyroid malignancy were ≥95 kPa for Emaximum, ≥85.2 kPa for Emean and ≥54.2 kPa for Eminimum [34]. However, Bhatia et al. [35] found that Emean of 34.5 kPa or higher was a significant predictor of malignancy with a sensitivity of 52.9% and specificity of 77.8%. Other studies suggested that SWE was useful in differentiating benign and malignant thyroid nodules when the cut-off level of 66 kPa and 65 kPa for Emaximum were used, respectively [15,36]. However, Szczepanek-Parulska et al. [37] found that the threshold value of 50 kPa for Emaximum was the most useful SWE parameter in the differentiation of benign and malignant nodules. Using the cut-off value of ≥53.2 kPa for Emaximum ≥ 34.5 kPa for Emean and ≥21.8 kPa for Eminimum, Duan et al. [33], found that SWE was superior to conventional GSU in identifying malignant nodules. Among different SWE indices, they found that Emean yielded the highest diagnostic accuracy (79.6%). Liu et al. [13] also reported that Emean ≥ 38.3 kPa was the most useful predictor among all SWE indices in differentiating benign and malignant thyroid nodule. In the present study, the optimal cut-off for Emaximum and Emean for distinguishing benign and malignant nodules were 67.3 and 23.1 kPa, respectively, which were consistent with the result of previous reports in Emaximum [15,36]. In our study, there was no significant difference in the Eminimum of benign and malignant nodules, which was different from previous studies [13,33,34]. The difference was due to the different methodologies used in the present and previous studies. Previous studies used ROI (Q-boxTM) with fixed size and placed it over the stiffer region of the nodule for the stiffness measurement. It involved subjective judgement of the operator in the placement of the ROI. However, the present study adjusted the size of the ROI so that it covered the entire nodule. The process did not involve operator’s judgement in which the ROI should be placed, and thus it was more objective. In addition, the method used in the present study allowed for a more comprehensive assessment of the nodule because the ROI covered the entire nodule in the image. Our result suggested that the tissue stiffness within malignant thyroid nodules are varied, with some areas are significantly stiffer than benign nodules (as demonstrated by significantly higher Emaximum and Emean in malignant nodules), whereas some areas have similar stiffness as benign nodules (as demonstrated by the similar Eminimum between benign and malignant nodules). This may be related to the uneven distribution of tumor cells within the nodule. Results of our study demonstrated that Emaximum and Emean are potential predictors for thyroid malignancy when using cut-off value of 67.3 kPa and 23.1 kPa, respectively. However, Eminimum has limited value in distinguishing benign and malignant nodules.
In the present study, we evaluated the diagnostic performance of GSU alone and a combination of GSU and SWE. Results showed that the overall diagnostic accuracy of GSU alone was 58.5% and it was increased to 80.2% and 78.4% when it combined with Emaximum (cut-off: 67.3 kPa) and Emean (cut-off: 23.1 kPa), respectively. Our results also demonstrated that when GSU combined with SWE, and the specificity increased from 46.4% to 83.3% when using Emaximum and to 79.8% when using Emean. However, the sensitivity decreased from 96.3% to 70.4% and 74.1%, respectively. The finding was different from previous reports. In Dobruch-Sobczak et al. [38], they reported that the combination of GSU with SWE did not significantly improve the diagnostic accuracy of malignant thyroid nodules. In the other studies, there was no significant difference in the diagnostic accuracy of GSU alone and combination of GSU and SWE indices in distinguishing benign and malignant thyroid nodules, 86.3% and 87.2% [34]; 81% and 77.9% [39]; and, 60.3% and 60.3% [36], respectively. They also found that when combined GSU with SWE indices the specificity decreased but the sensitivity increased [34,36,39]. One previous study that analysed thyroid nodules in population of France and found that there was an increase in sensitivity from 77.1% to 97.1% however specificity was reduced from 58.0 to 55.3% when grey scale ultrasound was added to Emaximum 65 kPa [36]. Similar results were obtained by another investigation conducted on thyroid nodules in Korean population and found that sensitivity was improved from 92.9% to 95% while specificity was reduced from 60.8% to 56.7% on addition of Eminimum 85.2 kPa. with grey scale ultrasound [34]. Another study conducted in China found an increase in sensitivity from 76.2% to 87.1%, while specificity was reduced from 83.0 to 73.9 kPa on addition of grey scale ultrasound to Eminimum 39.3 kPa [39]. The different result of previous reports and the present study was due to the different criteria in determining malignant thyroid nodules when combined with GSU with SWE. In previous studies, thyroid nodules were malignant when they either had one or more malignant grey scale sonographic features or had a SWE index value greater than the cut-off. This criterion increased the sensitivity by having more true-positive findings, but it also increased the number of false-positive findings leading to decreased specificity [34,36,38,39]. Since the extent of changes of specificity and sensitivity were similar, there was no significant improvement in the overall diagnostic accuracy when GSU combined with SWE in previous reports. However, in the present study, we considered thyroid nodules to be malignant when they had both malignant grey scale sonographic features (one or more features) and had a SWE index value greater than the cut-off. Using this assessment criterion, there was a substantial improvement in specificity with a moderate decrease in sensitivity leading to a significant improvement in the overall accuracy. Using the samples of the present study, the overall diagnostic accuracy was improved from 58.5% to 80.2% in Emaximum and to 78.3% in Emean. Although the diagnostic accuracy of combining GSU and SWE in our study was similar to those in previous studies, our study demonstrated substantial improvement in in the diagnostic accuracy after combining the two techniques, whereas previous studies showed no significance difference or decreased in the diagnostic accuracy [34,36,39,40] (Table 4). The similar accuracy of the present study (78.3–80.2%) and previous studies (60.3–87.2%) was probably due to the use of different samples of the patients in the studies. However, we believe that the method that we proposed in our study can enhance the diagnostic accuracy in differential diagnosis of thyroid nodules.
In routine clinical thyroid ultrasound examination, operators should consider examining the internal cervical chain to identify any metastatic cervical lymph nodes when malignant thyroid nodule is found. Ultrasound is a useful imaging tool to assess cervical lymph nodes. Metastatic cervical lymph nodes from papillary thyroid carcinoma are usually hyperechoic when compared to adjacent muscles, round in shape, without echogenic hilus, and have punctate calcification [41]. In addition, the combination of ultrasound and computed tomography can help to predict extrathyroidal extension [42], and fluorodeoxyglucose positron emission tomography/computed tomography (FDG-PET/CT) scans should be considered for detecting metastases in post-operative patients with aggressive histology of differentiated thyroid cancer [43].
In the present study, there were 84 benign and 27 malignant thyroid nodules. The calculated power of this sample size in evaluating the performance of GSU and SWE indices in distinguishing benign and malignant thyroid nodules ranged between 0.874 and 0.999 (G*Power version 3.1.9.2, Düsseldorf, Germany). There were limitations in the present study. The majority of the malignant thyroid nodules were papillary thyroid cancer, and thus we did not evaluate the difference of SWE indices among different types of malignant thyroid nodules. Future studies with larger sample size of various types of thyroid cancer can be conducted to investigate the value of SWE in differential diagnosis of different types of thyroid cancer. We did not evaluate the intra-operator and inter-operator reliability in SWE measurement of thyroid nodule stiffness. However, a previous study reported that SWE has satisfactory intra-operator (0.65–0.78) and inter-operator (0.72–0.77) reliability in the evaluation of neck lesions [35].

5. Conclusions

SWE indices (Emaximum and Emean) were independent predictors for thyroid malignancy. Combining GSU with SWE indices (using a cutoff of ≥67.3 kPa and ≥23.1 kPa for Emaximum and Emean respectively) can improve the overall diagnostic accuracy in distinguishing benign and malignant thyroid nodules. SWE is a useful adjunct to GSU in the assessment of thyroid nodules.

Acknowledgments

This study was supported by a research grant from The Hong Kong Polytechnic University, Hong Kong (RU55).

Author Contributions

All authors (listed below) have made substantial contribution in this study. M.Y. provided the concept and design of the study. He performed the thyroid ultrasound examinations, and edited the manuscript. F.N.B. performed image and data analyses and wrote the manuscript. S.Y.W.L. helped in recruiting the patients and edited the manuscript. H.-C.L. contributed in performing the inter-observer study. H.K.W.L. and S.-P.Y. revised the manuscript for intellectual content.

Conflicts of Interest

The authors declare no conflict of interest. The founding sponsors had no role in the design of the study; in the collection, analyses, or interpretation of data; in the writing of the manuscript, and in the decision to publish the results.

References

  1. Kim, S.-Y.; Kim, E.-K.; Moon, H.J.; Yoon, J.H.; Kwak, J.Y. Application of texture analysis in the differential diagnosis of benign and malignant thyroid nodules: Comparison with gray-scale ultrasound and elastography. Am. J. Roentgenol. 2015, 205, W343–W351. [Google Scholar] [CrossRef] [PubMed]
  2. Kwak, J.Y.; Kim, E.-K. Ultrasound elastography for thyroid nodules: Recent advances. Ultrasonography 2014, 33, 75–82. [Google Scholar] [CrossRef] [PubMed]
  3. Chan, B.K.; Desser, T.S.; McDougall, I.R.; Weigel, R.J.; Jeffrey, R.B. Common and uncommon sonographic features of papillary thyroid carcinoma. J. Ultrasound Med. 2003, 22, 1083–1090. [Google Scholar] [CrossRef] [PubMed]
  4. Cooper, D.S.; Doherty, G.M.; Haugen, B.R.; Kloos, R.T.; Lee, S.L.; Mandel, S.J.; Mazzaferri, E.L.; McIver, B.; Pacini, F.; Schlumberger, M. Revised american thyroid association management guidelines for patients with thyroid nodules and differentiated thyroid cancer: The american thyroid association (ata) guidelines taskforce on thyroid nodules and differentiated thyroid cancer. Thyroid 2009, 19, 1167–1214. [Google Scholar] [CrossRef] [PubMed]
  5. Moon, W.-J.; Baek, J.H.; Jung, S.L.; Kim, D.W.; Kim, E.K.; Kim, J.Y.; Kwak, J.Y.; Lee, J.H.; Lee, J.H.; Lee, Y.H. Ultrasonography and the ultrasound-based management of thyroid nodules: Consensus statement and recommendations. Korean J. Radiol. 2011, 12, 1–14. [Google Scholar] [CrossRef] [PubMed]
  6. Cappelli, C.; Castellano, M.; Pirola, I.; Cumetti, D.; Agosti, B.; Gandossi, E.; Rosei, E.A. The predictive value of ultrasound findings in the management of thyroid nodules. QJM 2007, 100, 29–35. [Google Scholar] [CrossRef] [PubMed]
  7. Frates, M.C.; Benson, C.B.; Charboneau, J.W.; Cibas, E.S.; Clark, O.H.; Coleman, B.G.; Cronan, J.J.; Doubilet, P.M.; Evans, D.B.; Goellner, J.R. Management of thyroid nodules detected at us: Society of radiologists in ultrasound consensus conference statement 1. Radiology 2005, 237, 794–800. [Google Scholar] [CrossRef] [PubMed]
  8. Cappelli, C.; Castellano, M.; Pirola, I.; Gandossi, E.; De Martino, E.; Cumetti, D.; Agosti, B.; Rosei, E.A. Thyroid nodule shape suggests malignancy. Eur. J. Endocrinol. 2006, 155, 27–31. [Google Scholar] [CrossRef] [PubMed]
  9. Razavi, S.A.; Hadduck, T.A.; Sadigh, G.; Dwamena, B.A. Comparative effectiveness of elastographic and b-mode ultrasound criteria for diagnostic discrimination of thyroid nodules: A meta-analysis. Am. J. Roentgenol. 2013, 200, 1317–1326. [Google Scholar] [CrossRef] [PubMed]
  10. Sun, J.; Cai, J.; Wang, X. Real-time ultrasound elastography for differentiation of benign and malignant thyroid nodules. J. Ultrasound Med. 2014, 33, 495–502. [Google Scholar] [CrossRef] [PubMed]
  11. Haugen, B.R. 2015 american thyroid association management guidelines for adult patients with thyroid nodules and differentiated thyroid cancer: What is new and what has changed? Thyroid 2016, 26, 1–133. [Google Scholar] [CrossRef] [PubMed]
  12. Hong, Y.; Liu, X.; Li, Z.; Zhang, X.; Chen, M.; Luo, Z. Realeallines for adult patients with thyroid nodules and differentiated thyroidalignant thyroid nodules. J. Ultrasound Med. 2009, 28, 861–867. [Google Scholar] [CrossRef] [PubMed]
  13. Liu, B.-X.; Xie, X.-Y.; Liang, J.-Y.; Zheng, Y.-L.; Huang, G.-L.; Zhou, L.-Y.; Wang, Z.; Xu, M.; Lu, M.-D. Shear wave elastography versus real-time elastography on evaluation thyroid nodules: A preliminary study. Eur. J. Radiol. 2014, 83, 1135–1143. [Google Scholar] [CrossRef] [PubMed]
  14. Khoo, M.L.; Asa, S.L.; Witterick, I.J.; Freeman, J.L. Thyroid calcification and its association with thyroid carcinoma. Head Neck 2002, 24, 651–655. [Google Scholar] [CrossRef] [PubMed]
  15. Sebag, F.; Vaillant-Lombard, J.; Berbis, J.; Griset, V.; Henry, J.; Petit, P.; Oliver, C. Shear wave elastography: A new ultrasound imaging mode for the differential diagnosis of benign and malignant thyroid nodules. J. Clin. Endocrinol. Metab. 2010, 95, 5281–5288. [Google Scholar] [CrossRef] [PubMed]
  16. Cantisani, V.; Grazhdani, H.; Drakonaki, E.; D’Andrea, V.; Di Segni, M.; Kaleshi, E.; Calliada, F.; Catalano, C.; Redler, A.; Brunese, L. Strain us elastography for the characterization of thyroid nodules: Advantages and limitation. Int. J. Endocrinol. 2015, 2015, 908575. [Google Scholar] [CrossRef] [PubMed]
  17. Rosario, P.W.; Silva, A.L.D.; Borges, M.A.R.; Calsolari, M.R. Is doppler ultrasound of additional value to gray-scale ultrasound in differentiating malignant and benign thyroid nodules? Arch. Endocrinol. Metab. 2015, 59, 79–83. [Google Scholar] [CrossRef] [PubMed]
  18. Anil, G.; Hegde, A.; Chong, F. Thyroid nodules: Risk stratification for malignancy with ultrasound and guided biopsy. Cancer Imaging 2011, 11, 209–223. [Google Scholar] [PubMed]
  19. Khadra, H.; Bakeer, M.; Hauch, A.; Hu, T.; Kandil, E. Is vascular flow a predictor of malignant thyroid nodules? A meta-analysis. Gland Surg. 2016, 5, 576–582. [Google Scholar] [CrossRef] [PubMed]
  20. Moon, H.J.; Kwak, J.Y.; Kim, M.J.; Son, E.J.; Kim, E.-K. Can vascularity at power doppler us help predict thyroid malignancy? Radiology 2010, 255, 260–269. [Google Scholar] [CrossRef] [PubMed]
  21. Dudea, S.M.; Botar-Jid, C. Ultrasound elastography in thyroid disease. Med. Ultrason. 2015, 17, 74. [Google Scholar] [CrossRef] [PubMed]
  22. Kim, E.-K.; Park, C.S.; Chung, W.Y.; Oh, K.K.; Kim, D.I.; Lee, J.T.; Yoo, H.S. New sonographic criteria for recommending fine-needle aspiration biopsy of nonpalpable solid nodules of the thyroid. Am. J. Roentgenol. 2002, 178, 687–691. [Google Scholar] [CrossRef] [PubMed]
  23. Papini, E.; Guglielmi, R.; Bianchini, A.; Crescenzi, A.; Taccogna, S.; Nardi, F.; Panunzi, C.; Rinaldi, R.; Toscano, V.; Pacella, C.M. Risk of malignancy in nonpalpable thyroid nodules: Predictive value of ultrasound and color-doppler features. J. Clin. Endocrinol. Metab. 2002, 87, 1941–1946. [Google Scholar] [CrossRef] [PubMed]
  24. Benson, J.; Fan, L. Tissue Strain Analytics—A Complete Ultrasound Solution for Elastography; Siemens Healthcare White Paper; Global Siemens Headquaters: Munchen, Germany, 2012. [Google Scholar]
  25. Shiina, T. Jsum ultrasound elastography practice guidelines: Basics and terminology. J. Med. Ultrason. 2013, 40, 309–323. [Google Scholar] [CrossRef] [PubMed]
  26. Couade, M.; Pernot, M.; Prada, C.; Messas, E.; Emmerich, J.; Bruneval, P.; Criton, A.; Fink, M.; Tanter, M. Quantitative assessment of arterial wall biomechanical properties using shear wave imaging. Ultrasound Med. Biol. 2010, 36, 1662–1676. [Google Scholar] [CrossRef] [PubMed]
  27. Dighe, M.K. Elastography of thyroid masses. Ultrasound Clin. 2014, 9, 13–24. [Google Scholar] [CrossRef] [PubMed]
  28. Koperek, O.; Scheuba, C.; Puri, C.; Birner, P.; Haslinger, C.; Rettig, W.; Niederle, B.; Kaserer, K.; Chesa, P.G. Molecular characterization of the desmoplastic tumor stroma in medullary thyroid carcinoma. Int. J. Oncol. 2007, 31, 59–68. [Google Scholar] [CrossRef] [PubMed]
  29. Popli, M.B.; Rastogi, A.; Bhalla, P.; Solanki, Y. Utility of gray-scale ultrasound to differentiate benign from malignant thyroid nodules. Indian J. Radiol. Imaging 2012, 22, 63. [Google Scholar] [CrossRef] [PubMed]
  30. Wong, K.; Ahuja, A.T. Ultrasound of thyroid cancer. Cancer Imaging 2005, 5, 157. [Google Scholar] [CrossRef] [PubMed]
  31. Yoon, J.H.; Kwak, J.Y.; Moon, H.J.; Kim, M.J.; Kim, E.-K. The diagnostic accuracy of ultrasound-guided fine-needle aspiration biopsy and the sonographic differences between benign and malignant thyroid nodules 3 cm or larger. Thyroid 2011, 21, 993–1000. [Google Scholar] [CrossRef] [PubMed]
  32. Eilers, S.G.; LaPolice, P.; Mukunyadzi, P.; Kapur, U.; Wendel Spiczka, A.; Shah, A.; Saleh, H.; Adeniran, A.; Nunez, A.; Balachandran, I. Thyroid fineinedifferences between benign and malignant thyroid nodules 3 cm or larger. Cancer Cytopathol. 2014, 122, 745–750. [Google Scholar] [CrossRef] [PubMed]
  33. Duan, S.-B.; Yu, J.; Li, X.; Han, Z.-Y.; Zhai, H.-Y.; Liang, P. Diagnostic value of two-dimensional shear wave elastography in papillary thyroid microcarcinoma. Onco Targets Ther. 2016, 9, 1311. [Google Scholar] [PubMed]
  34. Park, A.Y.; Son, E.J.; Han, K.; Youk, J.H.; Kim, J.-A.; Park, C.S. Shear wave elastography of thyroid nodules for the prediction of malignancy in a large scale study. Eur. J. Radiol. 2015, 84, 407–412. [Google Scholar] [CrossRef] [PubMed]
  35. Bhatia, K.S.; Tong, C.S.; Cho, C.C.; Yuen, E.H.; Lee, Y.Y.; Ahuja, A.T. Shear wave elastography of thyroid nodules in routine clinical practice: Preliminary observations and utility for detecting malignancy. Eur. Radiol. 2012, 22, 2397–2406. [Google Scholar] [CrossRef] [PubMed]
  36. Veyrieres, J.-B.; Albarel, F.; Lombard, J.V.; Berbis, J.; Sebag, F.; Oliver, C.; Petit, P. A threshold value in shear wave elastography to rule out malignant thyroid nodules: A reality? Eur. J. Radiol. 2012, 81, 3965–3972. [Google Scholar] [CrossRef] [PubMed]
  37. Szczepanek-Parulska, E.; Woliński, K.; Stangierski, A.; Gurgul, E.; Biczysko, M.; Majewski, P.; Rewaj-Łosyk, M.; Ruchała, M. Comparison of diagnostic value of conventional ultrasonography and shear wave elastography in the prediction of thyroid lesions malignancy. PLoS ONE 2013, 8, e81532. [Google Scholar] [CrossRef] [PubMed]
  38. Dobruch-Sobczak, K.; Zalewska, E.B.; Gumińska, A.; Słapa, R.Z.; Mlosek, K.; Wareluk, P.; Jakubowski, W.; Dedecjus, M. Diagnostic performance of shear wave elastography parameters alone and in combination with conventional b-mode ultrasound parameters for the characterization of thyroid nodules: A prospective, dual-center study. Ultrasound Med. Biol. 2016, 42, 2803–2811. [Google Scholar] [CrossRef] [PubMed]
  39. Liu, B.; Liang, J.; Zheng, Y.; Xie, X.; Huang, G.; Zhou, L.; Wang, W.; Lu, M. Two-dimensional shear wave elastography as promising diagnostic tool for predicting malignant thyroid nodules: A prospective single-centre experience. Eur. Radiol. 2015, 25, 624–634. [Google Scholar] [CrossRef] [PubMed]
  40. Dong, M.-J.; Liu, Z.-F.; Zhao, K.; Ruan, L.-X.; Wang, G.-L.; Yang, S.-Y.; Sun, F.; Luo, X.-G. Value of 18f-fdg-pet/pet-ct in differentiated thyroid carcinoma with radioiodine-negative whole-body scan: A meta-analysis. Nucl. Med. Commun. 2009, 30, 639–650. [Google Scholar] [CrossRef] [PubMed]
  41. Ahuja, A.; Chow, L.; Chick, W.; King, W.; Metreweli, C. Metastatic cervical nodes in papillary carcinoma of the thyroid: Ultrasound and histological correlation. Clin. Radiol. 1995, 50, 229–231. [Google Scholar] [CrossRef]
  42. Lee, D.Y.; Kwon, T.-K.; Sung, M.-W.; Kim, K.H.; Hah, J.H. Prediction of extrathyroidal extension using ultrasonography and computed tomography. Int. J. Endocrinol. 2014, 2014, 351058. [Google Scholar] [CrossRef] [PubMed]
  43. Nascimento, C.; Borget, I.; Al Ghuzlan, A.; Deandreis, D.; Hartl, D.; Lumbroso, J.; Berdelou, A.; Lepoutre-Lussey, C.; Mirghani, H.; Baudin, E. Postoperative fluorine-18-fluorodeoxyglucose positron emission tomography/computed tomography: An important imaging modality in patients with aggressive histology of differentiated thyroid cancer. Thyroid 2015, 25, 437–444. [Google Scholar] [CrossRef] [PubMed]
Applsci 07 01103 g001 550
Figure 1. Transverse sonogram showing a papillary carcinoma in the right thyroid lobe of a 43-year-old patient (arrows). The tumor is hypoechoic when compare to the adjacent thyroid parenchyma, and has multiple microcalcifications (arrowheads).
Figure 1. Transverse sonogram showing a papillary carcinoma in the right thyroid lobe of a 43-year-old patient (arrows). The tumor is hypoechoic when compare to the adjacent thyroid parenchyma, and has multiple microcalcifications (arrowheads).
Applsci 07 01103 g001
Applsci 07 01103 g002 550
Figure 2. Longitudinal sonogram showing a papillary carcinoma in the left thyroid lobe of a 51-year-old patient (arrows). The nodule appeared hypoechoic, ill-defined, and had multiple microcalcifications (arrowheads).
Figure 2. Longitudinal sonogram showing a papillary carcinoma in the left thyroid lobe of a 51-year-old patient (arrows). The nodule appeared hypoechoic, ill-defined, and had multiple microcalcifications (arrowheads).
Applsci 07 01103 g002
Applsci 07 01103 g003 550
Figure 3. Transverse elastogram (upper) and grey scale sonogram (lower) of a benign nodule in the right thyroid lobe of a 37-year-old patient. The value of elasticity indices (Emaximum = 32 kPa and Emean = 17.6 kPa) were lower than the respective cut-off values reported in the present study.
Figure 3. Transverse elastogram (upper) and grey scale sonogram (lower) of a benign nodule in the right thyroid lobe of a 37-year-old patient. The value of elasticity indices (Emaximum = 32 kPa and Emean = 17.6 kPa) were lower than the respective cut-off values reported in the present study.
Applsci 07 01103 g003
Applsci 07 01103 g004 550
Figure 4. Longitudinal elastogram (upper) and grey scale sonogram (lower) of a papillary carcinoma in the right thyroid lobe of a 57-year-old patient. The value of elasticity indices (Emaximum = 82.5 kPa and Emean = 53.8 kPa) were higher than the respective cut-off values reported in the present study.
Figure 4. Longitudinal elastogram (upper) and grey scale sonogram (lower) of a papillary carcinoma in the right thyroid lobe of a 57-year-old patient. The value of elasticity indices (Emaximum = 82.5 kPa and Emean = 53.8 kPa) were higher than the respective cut-off values reported in the present study.
Applsci 07 01103 g004
Applsci 07 01103 g005 550
Figure 5. Receiver operating characteristic (ROC) curves used to determine the optimal cut-off level of Emaximum and Emean in distinguishing benign and malignant thyroid nodules. Area under the curve, AUC, of Emean and Emaximum were 0.71 and 0.785, respectively.
Figure 5. Receiver operating characteristic (ROC) curves used to determine the optimal cut-off level of Emaximum and Emean in distinguishing benign and malignant thyroid nodules. Area under the curve, AUC, of Emean and Emaximum were 0.71 and 0.785, respectively.
Applsci 07 01103 g005
Applsci 07 01103 g006 550
Figure 6. Receiver operating characteristic (ROC) curves showing comparison between grey scale ultrasound features alone (usg. combine) and in combination with Emaximum (usg. Emax) and Emean (usg. Emean) in distinguishing benign and malignant thyroid nodules. Area under the curve, AUC, of usg. combine, usg. Emean, and usg. Emaximum were 0.714, 0.775 and 0.769 respectively.
Figure 6. Receiver operating characteristic (ROC) curves showing comparison between grey scale ultrasound features alone (usg. combine) and in combination with Emaximum (usg. Emax) and Emean (usg. Emean) in distinguishing benign and malignant thyroid nodules. Area under the curve, AUC, of usg. combine, usg. Emean, and usg. Emaximum were 0.714, 0.775 and 0.769 respectively.
Applsci 07 01103 g006
Table
Table 1. Grey scale sonographic features of benign and malignant thyroid nodules.
Table 1. Grey scale sonographic features of benign and malignant thyroid nodules.
Grey Scale Ultrasound FeaturesNumber of Nodules (Percentage)p-Value (95% Confidence Interval)
Benign (n = 84)Malignant (n = 27)
Microcalcification
Yes6 (7.1%)21 (77.8%)<0.05
No78 (92.9%)6 (22.2%)(0.12–0.49)
Tall/width ratio >1
Yes 11 (13.1%)16 (59.3%)<0.05
No73 (86.9%)11 (40.7%)(0.29–0.74)
Hypoechogenicity---
Yes28 (33.3%)25 (92.6%)<0.05
No56 (66.7%)2 (7.4%)(0.42–0.71)
Irregular margins
Yes14 (16.7%)15 (55.6%)<0.05
No70 (83.3%)12 (44.4%)(0.38–0.83)
Absent Halo sign
Yes74 (88.1%)26 (96.3%)>0.05
No10 (11.9%)1 (3.7%)(0.65–1.01)
Internal solid echotexture
Yes61 (72.6%)27 (100%)>0.05
No23 (27.4%)0 (0%)(0.6–0.8)
Table
Table 2. Diagnostic performance of grey scale ultrasound (GSU), shear wave elastography (SWE) indices and combination of GSU and SWE in evaluation of thyroid nodules.
Table 2. Diagnostic performance of grey scale ultrasound (GSU), shear wave elastography (SWE) indices and combination of GSU and SWE in evaluation of thyroid nodules.
Ultrasound TechniquesSensitivity (%)Specificity (%)PPV (%)NPV (%)Accuracy (%)AUC
GSU96.346.436.697.558.50.714
Emax (67.3 kPa)70.470.243.288.170.30.785
Emean (23.1 kPa)74.166.741.788.968.50.710
GSU + Emax (67.3 kPa)70.483.357.689.780.20.769
GSU + Emean (23.1 kPa)74.179.854.190.578.40.775
GSU, grey scale ultrasound, PPV, positive predictive value; NPV, negative predictive value, AUC, area under the curve.
Table
Table 3. Shear wave elastography measurement of benign and malignant thyroid nodules.
Table 3. Shear wave elastography measurement of benign and malignant thyroid nodules.
SWE IndicesMedian ± 1 Standard Errorp-Value (95% Confidence Interval)
BenignMalignant
Emax50.3 ± 3.185.2 ± 8.1<0.05 (50.9–63.5; 73.0–106.1)
Emean20.2 ± 1.026.6 ± 2.5<0.05 (19.5–23.5; 23.5–33.9)
Emin3.9 ± 0.63.8 ± 1.2>0.05 (4.2–6.4; 3.7–8.8)
Table
Table 4. Differences in diagnostic accuracy between the results of the present and previous studies to highlight the significant improvement achieved in the current study.
Table 4. Differences in diagnostic accuracy between the results of the present and previous studies to highlight the significant improvement achieved in the current study.
Studies Involved and SWE Index UsedDiagnostic Accuracy (%)Difference in Diagnostic Accuracy (%)
GSU AloneGSU + SWE
Present study
Emaximum (67.3 kPa)58.580.221.7
Emean (23.1 kPa)58.578.319.8
Veyrieres et al. [36]
Emaximum (65 kPa)8177.9−3.1
Park et al. [34]
Eminimum (85.2 kPa)86.387.20.9
Liu et al. [39]
Eminimum (39.3 kPa)60.360.30

Share and Cite

MDPI and ACS Style

Baig, F.N.; Liu, S.Y.W.; Lam, H.-C.; Yip, S.-P.; Law, H.K.W.; Ying, M. Shear Wave Elastography Combining with Conventional Grey Scale Ultrasound Improves the Diagnostic Accuracy in Differentiating Benign and Malignant Thyroid Nodules. Appl. Sci. 2017, 7, 1103. https://doi.org/10.3390/app7111103

AMA Style

Baig FN, Liu SYW, Lam H-C, Yip S-P, Law HKW, Ying M. Shear Wave Elastography Combining with Conventional Grey Scale Ultrasound Improves the Diagnostic Accuracy in Differentiating Benign and Malignant Thyroid Nodules. Applied Sciences. 2017; 7(11):1103. https://doi.org/10.3390/app7111103

Chicago/Turabian Style

Baig, Faisal N., Shirley Y. W. Liu, Hoi-Chun Lam, Shea-Ping Yip, Helen K. W. Law, and Michael Ying. 2017. "Shear Wave Elastography Combining with Conventional Grey Scale Ultrasound Improves the Diagnostic Accuracy in Differentiating Benign and Malignant Thyroid Nodules" Applied Sciences 7, no. 11: 1103. https://doi.org/10.3390/app7111103

Note that from the first issue of 2016, this journal uses article numbers instead of page numbers. See further details here.

Article Metrics

Back to TopTop