Improving GAN Learning Dynamics for Thyroid Nodule Segmentation


      Thyroid nodules are lesions requiring diagnosis and follow-up. Tools for detecting and segmenting nodules can help physicians with this diagnosis. Besides immediate diagnosis, automated tools can also enable tracking of the probability of malignancy over time. This paper demonstrates a new algorithm for segmenting thyroid nodules in ultrasound images. The algorithm combines traditional supervised semantic segmentation with unsupervised learning using GANs. The hybrid approach has the potential to upgrade the semantic segmentation model’s performance, but GANs have the well-known problems of unstable learning and mode collapse. To stabilize the training of the GAN model, we introduce the concept of closed-loop control of the gain on the loss output of the discriminator. We find gain control leads to smoother generator training and avoids the mode collapse that typically occurs when the discriminator learns too quickly relative to the generator. We also find that the combination of the supervised and unsupervised learning styles encourages both low-level accuracy and high-level consistency. As a test of the concept of controlled hybrid supervised and unsupervised semantic segmentation, we introduce a new model named the StableSeg GAN. The model uses DeeplabV3+ as the generator, Resnet18 as the discriminator, and uses PID control to stabilize the GAN learning process. The performance of the new model in terms of IoU is better than DeeplabV3+, with mean IoU of 81.26% on a challenging test set. The results of our thyroid nodule segmentation experiments show that StableSeg GANs have flexibility to segment nodules more accurately than either comparable supervised segmentation models or uncontrolled GANs.


      To read this article in full you will need to make a payment

      Purchase one-time access:

      Academic & Personal: 24 hour online accessCorporate R&D Professionals: 24 hour online access
      One-time access price info
      • For academic or personal research use, select 'Academic and Personal'
      • For corporate R&D use, select 'Corporate R&D Professionals'


      Subscribe to Ultrasound in Medicine and Biology
      Already a print subscriber? Claim online access
      Already an online subscriber? Sign in
      Institutional Access: Sign in to ScienceDirect


        • Arjovsky M.
        • Chintala S.
        • Bottou L.
        Wasserstein generative adversarial networks.
        in: Precup D. Teh Y.W. Proceedings of the 34th International Conference on Machine Learning. Proceedings of Machine Learning Research. volume 70. PMLR, 2017: 214-223
        • Badrinarayanan V.
        • Kendall A.
        • Cipolla R.
        SegNet: A deep convolutional Encoder-Decoder architecture for image segmentation.
        IEEE Trans. Pattern Anal. Mach. Intell. 2017; 39: 2481-2495
      1. D. Bank, N. Koenigstein, R. Giryes, Autoencoders (2020). arXiv:2003.05991. 2003.05991

      2. A. Bochkovskiy, C.-Y. Wang, H.-Y. M. Liao, YOLOv4: Optimal speed and accuracy of object detection (2020). arXiv:2004.10934. 2004.10934

        • Bomeli S.R.
        • LeBeau S.O.
        • Ferris R.L.
        Evaluation of a thyroid nodule.
        Otolaryngol. Clin. North Am. 2010; 43
      3. 229–38, vii
      4. L.-C. Chen, G. Papandreou, I. Kokkinos, K. Murphy, A.L. Yuille, Semantic image segmentation with deep convolutional nets and fully connected CRFs (2014).. arXiv:1412.7062. 1412.7062

      5. L.-C. Chen, G. Papandreou, F. Schroff, H. Adam, Rethinking atrous convolution for semantic image segmentation (2017).. arXiv:1706.05587. 1706.05587

      6. L.-C. Chen, Y. Zhu, G. Papandreou, F. Schroff, H. Adam, Encoder-decoder with atrous separable convolution for semantic image segmentation (2018).. arXiv:1802.02611. 1802.02611

        • Cheplygina V.
        • de Bruijne M.
        • Pluim J.P.
        Not-so-supervised: A survey of semi-supervised, multi-instance, and transfer learning in medical image analysis.
        Medical Image Analysis. 2019; 54: 280-296
      7. Q. Dou, C. Ouyang, C. Chen, H. Chen, P.-A. Heng, Unsupervised cross-modality domain adaptation of ConvNets for biomedical image segmentations with adversarial loss(2018). arXiv:1804. 10916. 1804.10916

      8. I.J. Goodfellow, J. Pouget-Abadie, M. Mirza, B. Xu, D. Warde-Farley, S. Ozair, A. Courville, Y. Bengio, Generative adversarial networks(2014). arXiv:1406.2661. 1406.2661

        • Greenspan H.
        • van Ginneken B.
        • Summers R.M.
        Guest editorial deep learning in medical imaging: Overview and future promise of an exciting new technique.
        IEEE Trans. Med. Imaging. 2016; 35: 1153-1159
        • He K.
        • Gkioxari G.
        • Dollar P.
        • Girshick R.
        Mask R-CNN.
        2017 IEEE International Conference on Computer Vision (ICCV). IEEE, 2017: 2980-2988
        • He K.
        • Zhang X.
        • Ren S.
        • Sun J.
        Deep residual learning for image recognition.
        2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR). IEEE, 2016: 770-778
        • Hesamian M.H.
        • Jia W.
        • He X.
        • Kennedy P.
        Deep learning techniques for medical image segmentation: Achievements and challenges.
        Journal of Digital Imaging. 2019; 32: 582-596
        • Huang C.
        • Yu A.
        • Wang Y.
        • He H.
        Skin lesion segmentation based on mask R-CNN.
        International Conference on Virtual Reality and Visualization (ICVRV) 2020. IEEE, 2020: 63-67
        • Isola P.
        • Zhu J.-Y.
        • Zhou T.
        • Efros A.A.
        Image-to-image translation with conditional adversarial networks.
        2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR). IEEE, 2017: 5967-5976
      9. S. Jadon, A survey of loss functions for semantic segmentation(2020). arXiv:2006.14822. 2006.14822

        • Jones J.
        • Morgan M.
        Assessment of thyroid lesions (ultrasound). 2015;
        • Ker J.
        • Wang L.
        • Rao J.
        • Lim T.
        Deep learning applications in medical image analysis.
        IEEE Access. 2018; 6: 9375-9389
        • Litjens G.
        • Kooi T.
        • Bejnordi B.E.
        • Setio A.A.A.
        • Ciompi F.
        • Ghafoorian M.
        • van der Laak J.A.
        • van Ginneken B.
        • Sánchez C.I.
        A survey on deep learning in medical image analysis.
        Medical Image Analysis. 2017; 42: 60-88
        • Long J.
        • Shelhamer E.
        • Darrell T.
        Fully convolutional networks for semantic segmentation.
        2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR). IEEE, 2015: 3431-3440
      10. L. Mescheder, A. Geiger, S. Nowozin, Which training methods for GANs do actually converge? (2018). arXiv:1801.04406. 1801.04406

        • Pedraza L.
        • Vargas C.
        • Narváez F.
        • Durán O.
        • Muñoz E.
        • Romero E.
        An open access thyroid ultrasound image database.
        in: Romero E. Lepore N. 10th International Symposium on Medical Information Processing and Analysis. SPIE, 2015: 92870W
        • Ramesh K.
        • Kumar G.K.
        • Swapna K.
        • Datta D.
        • Rajest S.S.
        A review of medical image segmentation algorithms.
        EAI Endorsed Transactions on Pervasive Health and Technology. 2021; 7
      11. J. Redmon, A. Farhadi, YOLOv3: An incremental improvement(2018). arXiv:1804.02767. 1804.02767

      12. O. Ronneberger, P. Fischer, T. Brox, U-Net: Convolutional networks for biomedical image segmentation (2015). arXiv:1505.04597. 1505.04597

        • Salimans T.
        • Goodfellow I.
        • Zaremba W.
        • Cheung V.
        • Radford A.
        • Chen X.
        • Chen X.
        Improved techniques for training gans.
        in: Lee D. Sugiyama M. Luxburg U. Guyon I. Garnett R. Advances in Neural Information Processing Systems. volume 29. Curran Associates, Inc., 2016: 2234-2242
        • Szegedy C.
        • Liu W.
        • Jia Y.
        • Sermanet P.
        • Reed S.
        • Anguelov D.
        • Erhan D.
        • Vanhoucke V.
        • Rabinovich A.
        Going deeper with convolutions.
        2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR). IEEE, 2015: 1-9
        • Szegedy C.
        • Vanhoucke V.
        • Ioffe S.
        • Shlens J.
        • Wojna Z.
        Rethinking the inception architecture for computer vision.
        2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR). IEEE, 2016: 2818-2826
        • Thakur A.
        • Anand R.S.
        A local statistics based region growing segmentation method for ultrasound medical images.
        World Academy of Science, Engineering and Technology, International Journal of Medical, Health, Biomedical, Bioengineering and Pharmaceutical Engineering. 2007; 1: 564-569
      13. A. Tuysuzoglu, J. Tan, K. Eissa, A.P. Kiraly, M. Diallo, A. Kamen, Deep adversarial context-aware landmark detection for ultrasound imaging (2018).. arXiv:1805.10737. 1805.10737

        • Wang M.
        • Yuan C.
        • Wu D.
        • Zeng Y.
        • Zhong S.
        • Qiu W.
        Automatic segmentation and classification of thyroid nodules in ultrasound images with convolutional neural networks’.
        Springer International Publishing, Cham2021
      14. K. Xu, C. Li, J. Zhu, B. Zhang, Understanding and stabilizing GANs’ training dynamics with control theory (2019). arXiv:1909.13188. 1909.13188

      15. A. Yadav, S. Shah, Z. Xu, D. Jacobs, T. Goldstein, Stabilizing adversarial nets with prediction methods(2017). arXiv:1705.07364. 1705.07364

      16. D. Yang, D. Xu, S.K. Zhou, B. Georgescu, M. Chen, S. Grbic, D. Metaxas, D. Comaniciu, Automatic liver segmentation using an adversarial image-to-image network (2017). arXiv:1707.08037. 1707.08037

        • Yao S.
        • Yan J.
        • Wu M.
        • Yang X.
        • Zhang W.
        • Lu H.
        • Qian B.
        Texture synthesis based thyroid nodule detection from medical ultrasound images: Interpreting and suppressing the adversarial effect of in-place manual annotation.
        Front. Bioeng. Biotechnol. 2020; 8: 599
        • Ye H.
        • Hang J.
        • Chen X.
        • Di Xu
        • Chen J.
        • Ye X.
        • Zhang D.
        An intelligent platform for ultrasound diagnosis of thyroid nodules.
        Sci. Rep. 2020; 10: 13223
        • Yi X.
        • Walia E.
        • Babyn P.
        Generative adversarial network in medical imaging: A review.
        Medical Image Analysis. 2019; 58: 101552
        • Yu X.
        • Wang H.
        • Ma L.
        Detection of thyroid nodules with ultrasound images based on deep learning.
        Curr. Med. Imaging Rev. 2020; 16: 174-180
        • Zhou S.K.
        • Greenspan H.
        • Davatzikos C.
        • Duncan J.S.
        • Van Ginneken B.
        • Madabhushi A.
        • Prince J.L.
        • Rueckert D.
        • Summers R.M.
        A review of deep learning in medical imaging: Imaging traits, technology trends, case studies with progress highlights, and future promises.
        Proc. IEEE Inst. Electr. Electron. Eng. 2021; 109: 820-838
        • Zhu J.-Y.
        • Park T.
        • Isola P.
        • Efros A.A.
        Unpaired image-to-image translation using cycle-consistent adversarial networks.
        arXiv:1703.10593. 2017;
      17. 1703.10593