LUNG DISEASE CLASSIFICATION FROM RESPIRATORY SOUNDS USING CNN-TRANSFORMER KNOWLEDGE DISTILLATION

Abstract

Because of the availability of enormous volume of data and computing resources, deep neural networks have gained popularity as they find application in both industry and academia. For certain tasks where a large training dataset cannot be obtained, transfer learning has shown that complex models that are already trained on large datasets can be applied to specific tasks by fine-tuning. However, in the healthcare domain, there is always a shortage of publicly available data and resources. Also, large pre-trained models are complex and often have a higher memory requirement, making them difficult to deploy. To overcome this issue, Knowledge Distillation has been used widely in the healthcare domain. Knowledge Distillation has been successful in compressing large and complex models making them easier for deployment. To explore about the applications of Knowledge Distillation in Healthcare domain, we con ducted a Systematic Literature Review. We analyzed recent studies based on some research questions that we formulated. After our analysis, we found some research gaps. In this study, we explore the potential of utilizing transformers in a limited data setting. We propose a framework based on Knowledge Distillation to train transformers for classifying lung diseases using respiratory sounds. Our proposed framework combines the attributes of both Convolutional Neural Network (CNN) and Transformers i.e., the translation equivariance and inductive biases of CNNs with the ability of transformers to handle long range dependencies. We have used Wavegram-Logmel-CNN as the teacher and Audio Spectrogram Transformer (AST) as the student model. The results show that our proposed framework improves the accu racy of the Transformer model. We also discuss the future scope for further improvements.