Advertisement
Original Contribution| Volume 47, ISSUE 1, P114-124, January 2021

Diagnostic Performance of Ultrasound Computer-Aided Diagnosis Software Compared with That of Radiologists with Different Levels of Expertise for Thyroid Malignancy: A Multicenter Prospective Study

      Abstract

      The aim of the work described here was to evaluate the diagnostic performance of ultrasound thyroid computer-aided diagnosis (CAD) software. This multicenter prospective study included 494 patients (565 thyroid nodules) who underwent surgery or biopsy after ultrasonography at four hospitals from January 2019 to September 2019. The diagnostic performance metrics of different readers were calculated and compared with the pathologic results. The sensitivity of CAD was outstanding and was equivalent to that of a senior radiologist (90.51% vs. 88.47%, p > 0.05). The area under the curve of CAD was equivalent to that of a junior radiologist (0.748 vs. 0.739, p > 0.05). However, the specificity was only 49.63%, which was lower than those of the three radiologists (75.56%, 85.93% and 90.37% for the junior, intermediate and senior radiologists, respectively). The diagnostic performance of the junior radiologist was significantly improved with the aid of CAD (junior + CAD). The sensitivity and area under the curve of junior + CAD were improved from 72.20% to 89.93% and from 0.739 to 0.816, respectively (both p values <0.05), and the positive predictive value, negative predictive value and κ coefficient improved from 76.3% to 78.6%, 82.0% to 86.8% and 0.394 to 0.511, respectively. Though specificity slightly decreased from 75.56% to 73.33%, the difference was not statistically significant (p > 0.05). In general, the clinical application value of CAD is promising, and its instrumental value for junior radiologists is significant.

      Key Words

      To read this article in full you will need to make a payment

      Purchase one-time access:

      Academic & Personal: 24 hour online accessCorporate R&D Professionals: 24 hour online access
      One-time access price info
      • For academic or personal research use, select 'Academic and Personal'
      • For corporate R&D use, select 'Corporate R&D Professionals'

      Subscribe:

      Subscribe to Ultrasound in Medicine and Biology
      Already a print subscriber? Claim online access
      Already an online subscriber? Sign in
      Institutional Access: Sign in to ScienceDirect

      References

        • Acharya U.R.
        • Faust O.
        • Sree S.V.
        • Molinari F.
        • Suri J.S.
        ThyroScreen system: High resolution ultrasound thyroid image characterization into benign and malignant classes using novel combination of texture and discrete wavelet transform.
        Comput Methods Programs Biomed. 2012; 107: 233-241
        • Ardakani A.A.
        • Gharbali A.
        • Mohammadi A.
        Classification of benign and malignant thyroid nodules using wavelet texture analysis of sonograms.
        J Ultrasound Med. 2015; 34: 1983-1989
        • Balleyguier C.
        • Kinkel K.
        • Fermanian J.
        • Malan S.
        • Djen G.
        • Taourel P.
        • Helenon O.
        Computer-aided detection (CAD) in mammography: Does it help the junior or the senior radiologist?.
        Eur J Radiol. 2005; 54: 90-96
        • Caruso D.
        • Mazzaferri E.L.
        Fine needle aspiration biopsy in the management of thyroid nodules.
        Endocrinologist. 1991; 1: 194-202
      1. Chang KJ, Chen WH, Chen A, Chen CN, Ho MC, Tai HC, Wu MH, Tsai PW. Method for retrieving a tumor contour of an image processing system. 2013 [P]: China, CN 102156874 B. 2015-07-29.

        • Chang C.C.
        • Chen H.H.
        • Chang Y.C.
        • Yang M.Y.
        • Lo C.M.
        • Ko W.C.
        • Lee Y.F.
        • Liu K.L.
        • Chang R.F.
        Computer-aided diagnosis of liver tumors on computed tomography images.
        Comput Methods Programs Biomed. 2017; 145: 45-51
        • Chen K.Y.
        • Chen C.N.
        • Wu M.H.
        • Ho M.C.
        • Tai H.C.
        • Kuo W.H.
        • Huang W.C.
        • Wang Y.H.
        • Chen A.
        • Chang K.J.
        Computerized quantification of ultrasonic heterogeneity in thyroid nodules.
        Ultrasound Med Biol. 2014; 40: 2581-2589
        • Cheng J.Z.
        • Ni D.
        • Chou Y.H.
        • Qin J.
        • Tiu C.M.
        • Chang Y.C.
        • Huang C.S.
        • Shen D.
        • Chen C.M.
        Computer-aided diagnosis with deep learning architecture: Applications to breast lesions in US images and pulmonary nodules in CT scans.
        Sci Rep. 2016; 6: 1-13
        • Choi S.H.
        • Kim E.K.
        • Kwak J.Y.
        • Kim M.J.
        • Son E.J.
        Interobserver and intraobserver variations in ultrasound assessment of thyroid nodules.
        Thyroid. 2010; 20: 167-172
        • Cibas E.S.
        • Ali S.Z.
        The 2017 Bethesda System for Reporting Thyroid Cytopathology.
        Thyroid. 2017; 27: 1341-1346
        • Da Fang W.M.
        • Xu L.
        • Liu Y.
        • Ma X.
        • Lu H.
        A predictive model to distinguish papillary thyroid carcinomas from benign thyroid nodules using ultrasonographic features: A single-center, retrospective analysis.
        Med Sci Monit. 2019; 25: 9409
        • De Nicola H.
        • Szejnfeld J.
        • Logullo Â.F.
        • Wolosker Â.M.B.
        • Souza L.R.M.F.
        • Chiferi Jr, V.
        Flow pattern and vascular resistive index as predictors of malignancy risk in thyroid follicular neoplasms.
        J Ultrasound Med. 2005; 24: 897-904
        • DeLong E.R.
        • DeLong D.M.
        • Clarke-Pearson D.L.
        Comparing the areas under two or more correlated receiver operating characteristic curves: A nonparametric approach.
        Biometrics. 1988; 44: 837-845
        • Gao L.
        • Liu R.
        • Jiang Y.
        • Song W.
        • Wang Y.
        • Liu J.
        • Wang J.
        • Wu D.
        • Li S.
        • Hao A.
        Computer‐aided system for diagnosing thyroid nodules on ultrasound: A comparison with radiologist‐based clinical assessments.
        Head Neck. 2018; 40: 778-783
        • Gharib H.
        • Goellner J.R.
        Fine-needle aspiration biopsy of the thyroid: An appraisal.
        Ann Intern Med. 1993; 118: 282-289
        • Gharib H.
        • Papini E.
        • Garber J.R.
        • Duick D.S.
        • Harrell R.M.
        • Hegedüs L.
        • Paschke R.
        • Valcavi R.
        • Vitti P.
        American Association of Clinical Endocrinologists, American College of Endocrinology, and Associazione Medici Endocrinologi medical guidelines for clinical practice for the diagnosis and management of thyroid nodules—2016 update.
        Endocr Practice. 2016; 22: 1-60
        • Ha S.M.
        • Ahn H.S.
        • Baek J.H.
        • Ahn H.Y.
        • Chung Y.J.
        • Cho B.Y.
        • Park S.B.
        Validation of three scoring risk-stratification models for thyroid nodules.
        Thyroid. 2017; 27: 1550-1557
        • Jeong E.Y.
        • Kim H.L.
        • Ha E.J.
        • Park S.Y.
        • Cho Y.J.
        • Han M.
        Computer-aided diagnosis system for thyroid nodules on ultrasonography: Diagnostic performance and reproducibility based on the experience level of operators.
        Eur Radiol. 2019; 29: 1978-1985
        • Jung K.
        • Won Y.
        • Kong H.
        • Lee E.
        Community of Population-Based Regional Cancer Registries. Cancer statistics in Korea: Incidence, mortality, survival, and prevalence in 2015.
        Cancer Res Treat. 2018; 50: 303-316
        • Kim H.G.
        • Kwak J.Y.
        • Kim E.K.
        • Choi S.H.
        • Moon H.J.
        Man to man training: Can it help improve the diagnostic performances and interobserver variabilities of thyroid ultrasonography in residents?.
        Eur J Radiol. 2012; 81: e352-e356
        • Kim H.L.
        • Ha E.J.
        • Han M.
        Real-world performance of computer-aided diagnosis system for thyroid nodules using ultrasonography.
        Ultrasound Med Biol. 2019; 45: 2672-2678
        • Komatsu M.
        • Hanamura N.
        • Tsuchiya S.
        • Seki T.
        • Kuroda T.
        Preoperative diagnosis of the follicular variant of papillary carcinoma of the thyroid: discrepancy between image and cytologic diagnoses.
        Radiat Med. 1994; 12: 293-299
        • Kooi T.
        • Litjens G.
        • Van Ginneken B.
        • Gubern-Mérida A.
        • Sánchez C.I.
        • Mann R.
        • den Heeten A.
        • Karssemeijer N.
        Large scale deep learning for computer aided detection of mammographic lesions.
        Med Image Anal. 2017; 35: 303-312
        • Kwak J.Y.
        • Koo H.
        • Youk J.H.
        • Kim M.J.
        • Moon H.J.
        • Son E.J.
        • Kim E.K.
        Value of US correlation of a thyroid nodule with initially benign cytologic results.
        Radiology. 2010; 254: 292-300
        • Kwak J.Y.
        • Han K.H.
        • Yoon J.H.
        • Moon H.J.
        • Son E.J.
        • Park S.H.
        • Jung H.K.
        • Choi J.S.
        • Kim B.M.
        • Kim E.K.
        Thyroid Imaging Reporting and Data System for US features of nodules: A step in establishing better stratification of cancer risk.
        Radiology. 2011; 260: 892-899
        • Kwak J.Y.
        • Jung I.
        • Baek J.H.
        • Baek S.M.
        • Choi N.
        • Choi Y.J.
        • Jung S.L.
        • Kim E.K.
        • Kim J.A.
        • Kim J.H.
        Image reporting and characterization system for ultrasound features of thyroid nodules: multicentric Korean retrospective study.
        Korean J Radiol. 2013; 14: 110-117
        • Lee R.S.
        • Gimenez F.
        • Hoogi A.
        • Miyake K.K.
        • Gorovoy M.
        • Rubin D.L.
        A curated mammography data set for use in computer-aided detection and diagnosis research.
        Sci Data. 2017; 4170177
        • Lehman C.D.
        • Wellman R.D.
        • Buist D.S.
        • Kerlikowske K.
        • Tosteson A.N.
        • Miglioretti D.L.
        Diagnostic accuracy of digital screening mammography with and without computer-aided detection.
        JAMA Intern Med. 2015; 175: 1828-1837
        • Li G.
        • Wang L.
        • Lei J.
        • Song L.
        • Tang H.
        • Li Z.
        • Gong R.
        • Zhu J.
        Large-scale comparative analysis reveals a simple model to predict the prevalence of thyroid nodules.
        Risk Manag Healthc Policy. 2019; 12: 225
        • Lu Y.
        • Shi X.Q.
        • Zhao X.
        • Song D.
        • Li J.
        Value of computer software for assisting sonographers in the diagnosis of Thyroid Imaging Reporting and Data System grade 3 and 4 thyroid space‐occupying lesions.
        J Ultrasound Med. 2019; 38: 3291-3300
        • Marvasti N.B.
        • Yörük E.
        • Acar B.
        Computer-aided medical image annotation: Preliminary results with liver lesions in CT.
        IEEE J Biomed Health Informatics. 2017; 22: 1561-1570
        • McNemar Q.
        Note on the sampling error of the difference between correlated proportions or percentages.
        Psychometrika. 1947; 12: 153-157
        • Nayak A.
        • Kayal E.B.
        • Arya M.
        • Culli J.
        • Krishan S.
        • Agarwal S.
        • Mehndiratta A.
        Computer-aided diagnosis of cirrhosis and hepatocellular carcinoma using multi-phase abdomen CT.
        Int J Computer Assisted Radiol Surg. 2019; 14: 1341-1352
        • Nishio M.
        • Sugiyama O.
        • Yakami M.
        • Ueno S.
        • Kubo T.
        • Kuroda T.
        • Togashi K.
        Computer-aided diagnosis of lung nodule classification between benign nodule, primary lung cancer, and metastatic lung cancer at different image size using deep convolutional neural network with transfer learning.
        PloS One. 2018; 13e0200721
        • Ota H.
        • Ito Y.
        • Matsuzuka F.
        • Kuma S.
        • Fukata S.
        • Morita S.
        • Kobayashi K.
        • Nakamura Y.
        • Kakudo K.
        • Amino N.
        Usefulness of ultrasonography for diagnosis of malignant lymphoma of the thyroid.
        Thyroid. 2006; 16: 983-987
        • Ouyang F.S.
        • Guo B.L.
        • Ouyang L.Z.
        • Liu Z.W.
        • Lin S.J.
        • Meng W.
        • Huang X.Y.
        • Chen H.X.
        • Qiugen H.
        • Yang S.M.
        Comparison between linear and nonlinear machine-learning algorithms for the classification of thyroid nodules.
        Eur J Radiol. 2019; 113: 251-257
        • Park S.H.
        • Kim S.J.
        • Kim E.K.
        • Kim M.J.
        • Son E.J.
        • Kwak J.Y.
        Interobserver agreement in assessing the sonographic and elastographic features of malignant thyroid nodules.
        AJR Am J Roentgenol. 2009; 193: W416-W423
        • Park H.J.
        • Kim S.M.
        • La Yun B.
        • Jang M.
        • Kim B.
        • Jang J.Y.
        • Lee J.Y.
        • Lee S.H.
        A computer-aided diagnosis system using artificial intelligence for the diagnosis and characterization of breast masses on ultrasound: added value for the inexperienced breast radiologist.
        Medicine (Baltimore). 2019; 98: e14146
        • Pellegriti G.
        • Frasca F.
        • Regalbuto C.
        • Squatrito S.
        • Vigneri R.
        Worldwide increasing incidence of thyroid cancer: update on epidemiology and risk factors.
        J Cancer Epidemiol. 2013; 2013965212
      2. Perros P, Boelaert K, Colley S, Evans C, Evans RM, Gerrard BG, Gilbert J, Harrison B, Johnson SJ, Giles TE. British Thyroid Association Guidelines for the Management of Thyroid Cancer. 2014;81:1–122.

        • Pitoia F.
        • Miyauchi A.
        2015 American Thyroid Association guidelines for thyroid nodules and differentiated thyroid cancer and their implementation in various care settings.
        Thyroid. 2016; 26: 319-321
        • Reverter J.L.
        • Vázquez F.
        • Puig-Domingo M.
        Diagnostic performance evaluation of a computer-assisted imaging analysis system for ultrasound risk stratification of thyroid nodules.
        AJR Am J Roentgenol. 2019; 213: 169-174
        • Russ G.
        Risk stratification of thyroid nodules on ultrasonography with the French TI-RADS: Description and reflections.
        Ultrasonography. 2016; 35: 25
        • Seo H.
        • Na D.G.
        • Kim J.H.
        • Kim K.W.
        • Yoon J.W.
        Ultrasound-based risk stratification for malignancy in thyroid nodules: A four-tier categorization system.
        Eur Radiol. 2015; 25: 2153-2162
        • Silva M.
        • Schaefer-Prokop C.M.
        • Jacobs C.
        • Capretti G.
        • Ciompi F.
        • van Ginneken B.
        • Pastorino U.
        • Sverzellati N.
        Detection of subsolid nodules in lung cancer screening: Complementary sensitivity of visual reading and computer-aided diagnosis.
        Invest Radiol. 2018; 53: 441-449
        • Tessler F.N.
        • Middleton W.D.
        • Grant E.G.
        • Hoang J.K.
        • Berland L.L.
        • Teefey S.A.
        • Cronan J.J.
        • Beland M.D.
        • Desser T.S.
        • Frates M.C.
        ACR Thyroid Imaging, Reporting and Data System (TI-RADS): White paper of the ACR TI-RADS committee.
        J Am Coll Radiol. 2017; 14: 587-595
        • Yassin N.I.
        • Omran S.
        • El Houby E.M.
        • Allam H.
        Machine learning techniques for breast cancer computer aided diagnosis using different image modalities: A systematic review.
        Comput Methods Programs Biomed. 2018; 156: 25-45