ANOMALY-BASED NETWORK INTRUSION DETECTION SYSTEM USING MACHINE LEARNING ALGORITHMS

Abstract

The rapid advancement of technology not only simplifies life but also introduces nu- merous security challenges. Over the years, as the Internet has evolved, the frequency and sophistication of cyberattacks have increased significantly, targeting individuals, or- ganizations, and critical infrastructures. This growing threat underscores the vital need for robust security frameworks. Intrusion Detection Systems (IDS) play a crucial role in continuously monitoring network activity, identifying malicious behaviours, and miti- gating potential attacks in real time. Hence, anomaly-based network intrusion detection powered with machine learning techniques is proposed in this thesis to develop intelli- gent and adaptive IDS solutions, which are crucial for maintaining strong cybersecurity defences. This thesis aims to enhance the effectiveness of Intrusion Detection Systems by initially addressing the challenge of selecting the most relevant features from high- dimensional network traffic data. The presence of redundant and irrelevant features can lead to increased computational complexity and reduced detection accuracy. We proposed a three-phase network-based IDS to counter this issue, where we developed a dynamic mutual information-based genetic algorithm (DMI-GA), a novel feature se- lection technique designed to identify an optimal set of features. By integrating mutual information to measure feature relevance and a genetic algorithm to optimize selec- tion, DMI-GA enhances both the efficiency and accuracy of IDS models. Unlike many existing feature selection methods that evaluate each feature independently and fail to account for feature dependencies, our approach considers the relationships between fea- tures, leading to more informed selection and improved computational performance. This method not only reduces dimensionality but also ensures that the most significant features contribute to better attack detection. The high dimensionality of data degrades IDS performance, causing sparsity is- sues that obscure meaningful patterns. It also increases the risk of overfitting, making models learn noise instead of actual attack behaviours, reducing their ability to detect new threats. Also, since dataset quality is crucial for accurately detecting and classi- fying intrusions, the presence of highly imbalanced data, where benign network pack- ets significantly outnumber anomalous ones, can deteriorate classification performance. vi Thus, we developed another anomaly-based IDS in conjunction with machine learning techniques and a novel modified picture fuzzy clustering-based approach, mPicF C, on the dimensionality-reduced dataset. This approach incorporates an additional decision- making layer to handle uncertainty more effectively. By differentiating between partial membership and complete non-membership, it enables more precise classifications. The inclusion of refusal or hesitation degrees helps minimize bias in clustering, preventing uncertain data points from disproportionately influencing the results. Moreover, the proposed framework addresses the class imbalance by reducing bias toward the ma- jority class, using the Synthetic Minority Oversampling Technique (SMOTE), which ultimately improved the model’s accuracy. To address the growing need for real-time threat detection, we propose HIL-IDS, a real-time Intrusion Detection System based on a hybrid incremental learning approach. HIL-IDS continuously monitors network traffic, detects anomalies, and adapts to evolv- ing cyber threats with minimal latency. It integrates the Hoeffding Tree for incremental supervised learning, leveraging its efficiency in processing streaming data, and an en- semble of Isolation Forest and K-Means for unsupervised anomaly detection, effective in identifying novel attack patterns without prior labels. Confidence scores from the combination of these supervised and unsupervised models are evaluated to enhance the interpretability of the proposed framework. To maintain robustness against shifting data distributions, drift detection enables adaptation to emerging threats in real time. By combining multiple anomaly detection methods in an ensemble, HIL-IDS improves the likelihood of detecting diverse attack types. While hybrid intrusion detection systems exist, the integration of incremental learning in both supervised and unsupervised com- ponents allows HIL-IDS to dynamically adapt to evolving attacks and network traffic in real time, making it highly suitable for modern, dynamic network environments. The performance of the developed methods is compared with the various contempo- rary models on the sophisticated network traffic datasets using quantitative and statisti- cal sampling assessments. The empirical results, along with statistical tests, show the superiority of the proposed methods over the existing methods for intrusion detection. Overall, the findings establish a strong foundation for future research in developing more adaptive and intelligent intrusion detection systems, enhancing real-time threat detection, and improving the scalability and efficiency of cybersecurity solutions.