DESIGN AND ANALYSIS OF CLASSIFICATION FRAMEWORK FOR STREAMING DATA

Abstract

The rapid growth of digital technologies, sensor networks, and internet services generates massive amounts of streaming data that arrive sequentially and must be processed in real time under strict constraints. Its statistical properties often change over time due to environmental factors, user behavior, or system dynamics, causing concept drift. This challenges traditional machine learning models, which assume stationary data, resulting in degraded predictive performance on evolving data streams. Therefore, adaptive, scalable online machine learning frameworks are needed to learn incrementally, detect changes in patterns, and maintain performance in dynamic environments. This thesis explores robust classification for drift detection and adaptation, as well as anomaly-detection frameworks for streaming data. The research combines ensemble learning, drift detection, and online methods to create adaptive systems for evolving data. It explores various online frameworks in real-world streaming, aiding machine learning systems that process large data streams in real time while staying accurate, adaptable, and efficient. It begins with an adaptive ensemble classification framework to tackle concept drift, which can impair model performance. To address this, an error-weighted drift-adaptive ensemble framework improves classification with evolving data streams by combining multiple online classifiers, each built with adaptive random forests based on Hoeffding trees for incremental learning. It integrates multiple drift-detection techniques that monitor individual prediction errors to detect data shifts. When detected, the framework adapts by updating the ensemble and adjusting the learning rate. A dynamic error weighting strategy assigns weights based on real-time errors, giving more influence to better models. Experiments on real and synthetic data show that this approach significantly improves accuracy, precision, recall, and F1-score compared with other methods, demonstrating its effectiveness in handling both gradual and abrupt concept drift. vi Building on the adaptive classification framework, this work explores anomaly detection in streaming environments, where identifying cyber intrusions, fraud, or hazards in real time is crucial. Detecting anomalies is challenging due to unlabeled data, shifting patterns, and ongoing learning. A heterogeneous, multi framework stacked ensemble approach combines River and PySAD, offering various unsupervised algorithms for incremental learning. The ensemble uses stacking: base models analyze the data and generate anomaly scores, which are then fed into meta learners such as logistic regression and AdaBoost to produce final predictions. By integrating tree-based, density-based, isolation-based, and distance-based methods, detection improves across different anomalies. Experiments on benchmark datasets show the ensemble outperforms individual models and latest techniques, confirming that combining diverse frameworks enhances accuracy and robustness. This study assesses Delhi's air quality data using online anomaly detection algorithms to identify pollution events. Sensor data monitor changes in pollutant levels and deviations in AQI to detect abnormal conditions. Multiple models are tested; some perform well across different stations and pollutants. Results indicate that online machine learning improves real-time monitoring and detection of dangerous pollution episodes. The study emphasizes the potential of streaming anomaly-detection algorithms to enhance the responsiveness and reliability of environmental management systems. This research develops classification and anomaly-detection frameworks and evaluates online machine learning tools for streaming data. With the rise of real time analytics, several tools support incremental learning and streaming data, but their effectiveness varies in terms of design, efficiency, and compatibility. A systematic evaluation compares their performance on classification and anomaly detection tasks using accuracy, precision, recall, F1 Score, and ROC AUC. Results show that some frameworks better support streaming data through optimized incremental learning, aiding in tool selection. Additional experiments evaluate unsupervised anomaly detection algorithms in IoT environments with continuous sensor data, revealing their strengths and limitations. Ensemble and hybrid methods generally outperform individual models in complex, evolving data. This study explores adaptive machine learning for streaming analytics, combining ensemble methods, heterogeneous frameworks, and performance assessments. The approaches improve prediction, anomaly detection, and adaptability, advancing data stream mining and online learning. Applications include network security, finance, environmental sensing, healthcare, and smart cities. This work enables effective processing of continuous data, supporting next generation real-time machine learning systems.