TOWARDS ROBUST AND GENERALIZABLE MACHINE LEARNING-BASED NETWORK INTRUSION DETECTION: FROM ENSEMBLE BASELINES TO DEEP LEARNING METHODS FOR COMPLEX AND EVOLVING THREAT ENVIRONMENT

Abstract

Machine Learning Based NIDS (Network Intrusion Detection Systems) attain almost flawless performance on standardized benchmarks but there are three major problems with such systems which include sensitivity towards class imbalance, vulnerability to adversarial attacks, and lack of generalization to different network scenarios. This thesis addresses all three through a progressive three-stage research programme on the CICIDS2017, CSE-CICIDS2018, and TON IoT datasets. In the first stage, a reproducible IDS pipeline is constructed on CICIDS2017 using imputation, standard scaling, SMOTE balancing, and stratified splitting. Random Forest, XGBoost, and MLP are trained and evaluated under identical conditions. XGBoost achieves the best performance (AUC = 0.995, FNR = 0.05%), and feature importance analysis identifies a compact top 10 feature subset accounting for 71.4% of total model importance. In the second stage, the adversarial vulnerability of the baseline models is system atically evaluated. White box FGSM and PGD attacks on the MLP raise FNR from 0.30% to 33.58% at 𝜀 = 0.1 a 112 fold increase in missed attacks. Black box transfer attacks on Random Forest and XGBoost reveal a novel FNR saturation phenomenon: both tree models reach approximately 34% FNR at the minimum perturbation budget 𝜀 =0.01 and show no further degradation with increasing perturbation, demonstrating that non differentiable classifiers are not inherently robust against transfer based evasion. Adversarial training recovers 15.4 percentage points of adversarial accuracy at a clean accuracy cost of only 2.0%. In the third stage, three heterogeneous datasets are integrated through a unified preprocessing pipeline and a compact eight-feature shared representation. Four deep learning architectures, namely CNN, LSTM, GRU, and a lightweight Transformer, are evaluated on the merged two-million flow corpus. Under within-distribution evaluation, the GRU achieves the best result (94.23% accuracy, F1 = 0.8714), with all four architectures exceeding 91%, demonstrating that compact flow representations support reliable detection across heterogeneous network environments without dataset specific feature engineering. A leave-one-dataset-out evaluation further reveals that cross-domain F1 scores range from 0.45 to 0.69 when models are tested on entirely unseen network environments, establishing a reproducible baseline and confirming that distribution shift between enterprise and IoT traffic remains a fundamental open challenge.