APPLICATION OF MACHINE LEARNING TECHNIQUES FOR MODERN CYBER THREAT DETECTION

Abstract

The increasing sophistication of modern malware has rendered traditional signature based detection techniques less effective, especially against polymorphic and zero-day threats. In response, machine learning (ML) and deep learning (DL) methods have emerged as viable alternatives for identifying malicious behaviour through pattern recog nition. This thesis investigates the application of ML techniques in cyber threat detection through two key components: a critical review of recent literature, and the implemen tation of an improved multiclass malware classification system. The review analyzes twenty contemporary research papers, comparing approaches based on datasets, analysis techniques, model choices, and detection performance. Key limitations identified include class imbalance, feature redundancy, and challenges in handling complex or obfuscated malware. Building on these insights, a hybrid ensemble model is developed combining tree-based classifiers and a neural network meta-learner. The system incorporates ad vanced preprocessing techniques such as mutual information-based feature selection and SMOTE-based oversampling to improve learning from imbalanced data. The model is trained and evaluated on the CIC MalMem2022 dataset, covering fifteen malware fami lies and benign samples. Results show improved classification performance, particularly for underrepresented classes, with notable gains in macro-averaged precision and recall. This work demonstrates the potential of integrated ML pipelines for practical malware detection and suggests further exploration into explainable models, adaptive learning, and cross-platform generalization for future research.