DEVELOPMENT AND VALIDATION OF SOFTWARE QUALITY PREDICTION MODELS USING MACHINE LEARNING TECHNIQUES

Abstract

Software quality plays a pivotal role in ensuring the success of software projects, especially in today’s rapidly evolving technological landscape. As software systems grow more complex, predicting their quality, identifying potential defects, and foresee ing maintenance challenges have become critical tasks. Software Quality Prediction Models (SQPMs) serve as powerful tools to forecast software defects, allowing organi zations to take proactive measures in improving system reliability, reducing costs, and enhancing overall product quality. By leveraging Machine Learning (ML) techniques, SQPMs help automate the prediction process, providing decision-makers with valu able insights into which components or modules of software are more prone to defects or changes. This, in turn, enables effective resource allocation, improved maintenance strategies, and a reduction in post-release failures. The increasing demand for efficient software quality models has led to extensive research on improving the performance of these models. This thesis, titled ’Develop ment and Validation of Software Quality Prediction Models using Machine Learning Techniques’, aims to enhance the performance of prediction models by developing robust classifiers and tackling critical issues that hinder the effectiveness of learning algorithms. This research outlines several key objectives, including conducting sys tematic reviews on the classification algorithms and various factors affecting software quality prediction models. A detailed analysis of the data imbalance problem, parame ter tuning, feature selection, and techniques for handling outliers and multi-collinearity are explored. Additionally, the research focuses on the validation of open-source datasets and the proposal of new metrics for software quality assessment. The thesis covers multiple experiments and methodologies aimed at tackling these objectives. Each experiment involves the application of state-of-the-art techniques and algorithms in software defect categorization (SDC) and software defect prediction (SDP). The first area of study focuses on the categorization of software defects based on maintenance effort, change impact, and a combination of the two. Various machine learning algorithms, including the Multinomial Na¨ıve Bayes (NBM), ensemble learn ers ( Random Forest (RF), eXtreme Gradient Boosting (XGB), Adaptive Boosting (ADB), and Bagging (BAG)) and Convolutional Neural Networks (CNN) were applied to develop software defect categorization (SDC) models. These models are utilized to categorize software defects into low, medium, and high categories based on three key defect attributes - maintenance effort, change impact, and their combined ef fect. Experiments were conducted on five Android applications - Bluetooth, Browser, Calendar, Camera, and MMS with the Top10, Top25, Top50, and Top100 relevant keywords extracted from the defect reports through text mining. The results, validated using AUC values, indicate acceptable predictability for defects categorized in various categories. Models based on combined approach demonstrated better performance than those built using only change impact and remain competitive with those built using only maintenance effort. Multinomial Na¨ıve Bayes (NBM) and Convolutional Neural Networks (CNNs) can be effectively used for software defect categorization. Random Forest emerged as the most effective ensemble technique, followed by Bag ging and AdaBoost. These models provide valuable insights for software practitioners in terms of effort estimation and resource management, offering a practical solution for addressing high-category defects that demand significant developer or tester effort. This thesis examines the application of parameter tuning techniques to software quality prediction models. Through a systematic literature review of 31 primary studies, this review identifies and analyzes various parameter tuning methods. A detailed analysis of these studies revealed that tuned models consistently demonstrated improved predictive performance and stability over untuned models. Among the most effective parameter tuning techniques identified were grid search, differential evolution, genetic algorithm-based approaches, and hybrid methods. Classification algorithms such as Support Vector Machine (SVM), k-Nearest Neighbor (KNN), Random Forest (RF), Neural Networks (NN), and Classification and Regression Trees (CART) were frequently subjected to parameter tuning, with notable sensitivity to hyperparameter adjustments. On the other hand, algorithms like Linear Regression, Regression Trees (RTs), and Bagging with RTs exhibited lower sensitivity to parameter tuning. The results also showed that parameter tuning significantly enhanced the performance of underperforming classifiers, affecting their ranking. The thesis offers practical recommendations for software practitioners, advocating for the use of tuning methods in predictive modeling to ensure optimal performance. A major challenge in predictive modeling is handling imbalanced datasets in software defect prediction (SDP). This thesis explores two approaches to address this challenge with focus on neural networks: (1) Data-Level Approach: This approach employed four oversampling techniques (ROS, SMOTE, BL-SMOTE, ADASYN), six undersampling techniques (RUS, CC, NM, CNN, TL, ENN), and two hybrid techniques combining oversampling and undersampling (SMOTE-TL, SMOTE-ENN) applied across six open-source software datasets. A total of 78 ANN-based defect prediction models were developed using these techniques, and stratified 10-fold cross validation was performed to ensure robust validation. The performance was assessed using AUC, G-Mean, and Balance metrics. The study found that oversampling and hy brid techniques, especially SMOTE-ENN, significantly enhanced model performance. Statistical analysis further confirmed that SMOTE-ENN, BL-SMOTE, SMOTE-TL, ROS, and SMOTE were the best-performing techniques, whereas models without resampling and those using undersampling ranked poorly. (2) Algorithm-Level Ap proach: A Weighted Loss Function for Neural Networks (WL-NN) was introduced to address data imbalance during model training. Four models were developed: NN on imbalanced data, WL-NN on imbalanced data, NN on balanced data via SMOTE-ENN (NN + SMOTE-ENN), and WL-NN on balanced data via SMOTE-ENN (WL-NN + SMOTE-ENN), resulting in 88 defect prediction models. The results demonstrated that WL-NN significantly improved model performance, with WL-NN + SMOTE ENN showing a 27% improvement compared to other models. This combination outperformed all approaches, establishing that weighted loss functions combined with data resampling are highly effective in addressing imbalanced data issues in SDP. Feature selection plays a vital role in improving the performance of SQPMs by eliminating irrelevant or redundant features from the dataset. This thesis focused on evaluating the effectiveness of swarm intelligence techniques - Ant Colony Optimiza tion (ACO), Cuckoo Search (CS), and Crow Search (CRS) - for feature selection in software defect prediction, comparing them to traditional filter-based methods such as Chi-Square (CHI2) and Information Gain (IG). Using 22 datasets from the AEEEM, JIRA, and PROMISE repositories, the results demonstrated that Cuckoo Search (CS) consistently outperformed the other methods, achieving the highest AUC values across most datasets and classifiers. Crow Search (CRS) also performed well, often rank ing just behind CS, especially when combined with classifiers like Support Vector Machine (SVM) and Random Forest (RF). In contrast, Ant Colony Optimization (ACO) showed mixed results, delivering strong performance in some cases but lacking consistency compared to CS and CRS. Overall, the study highlights the superior performance of swarm intelligence techniques, with Cuckoo Search (CS) and Crow Search (CRS) emerging as promising approaches to enhance defect prediction models, especially when integrated with advanced classifiers like RF and SVM. This thesis provides a comprehensive review of multi-collinearity in software qual ity prediction, a critical issue affecting the reliability, maintainability, and efficiency of predictive models. Through a detailed analysis of the literature, the paper highlights several challenges posed by multi-collinearity, such as uncertainty in predictor effects, overfitting, and reduced generalizability of software quality models. To address these challenges, the review explores various mitigation strategies, including traditional methods like principal component analysis (PCA), regularization techniques (ridge and lasso regression), stepwise regression, and variance inflation factor (VIF) thresh- olding. It also discusses more recent advancements such as hybrid PCA-regularization approaches, sparse partial least squares (SPLS), and modern machine learning tech niques, including Ensemble Learning and Deep Learning approaches. Among these, PCA, ridge regression, and lasso regression are identified as the most commonly employed techniques to combat multi-collinearity. This thesis proposes a comprehensive metric suite tailored for evaluating event driven software systems, which are increasingly prevalent due to their capacity to manage complex and asynchronous interactions. The study emphasizes the distinctive ness of event-driven systems compared to structured and object-oriented paradigms, underscoring the need for specialized metrics. The proposed metrics are categorized into key areas, including event structure, event dependency, event performance, event complexity, event synchronization, and event reliability. Each metric is thoroughly defined to provide a standardized, objective framework for assessing the unique characteristics and behavior of event-driven systems. In summary, this thesis contributes significantly to the field of software quality predictive modeling by addressing key issues such as imbalanced data, parameter tuning, feature selection, and algorithm optimization. The developed models and techniques have demonstrated their potential to improve the accuracy and efficiency of predictive models, thereby supporting better decision-making in software maintenance and quality assurance.