RESEARCH TRENDS IN TEXT SUMMARIZATION WITH A FOCUS ON FINE-TUNING INSTRUCTION-TUNED LANGUAGE MODELS

Abstract

Text summarisation has emerged as a crucial area within Natural Language Processing, driven by the increasing need to process and condense vast amounts of textual data. This thesis presents a twofold investigation into the domain: (1) a comprehensive bibliometric analysis of global research trends in text summarisation, and (2) a performance-focused study on hyperparameter tuning for domain-specific text generation using transformer-based models. The first study employs bibliometric methods, visual analytics, and critical review to trace the evolution of text summarisation across decades, highlighting key contributors, publication trends, influential methodologies, and emerging directions such as multilingual and multimodal summarisation. This exploration not only contextualizes the field’s growth but also identifies research gaps and strategic opportunities for innovation. Building upon these insights, the second study investigates the practical challenges of adapting instruction-tuned models for domain-specific summarisation. Using the BillSum legal dataset, the performance of two models—BART-Large and FLAN-T5 Base—was evaluated. Initial experiments revealed that FLAN-T5 underperformed significantly compared to BART-Large. Traditional strategies such as data augmentation and cross-domain pretraining failed to yield improvements. However, targeted hyperparameter optimization led to substantial gains, emphasizing the critical role of configuration over model architecture alone. Together, these studies offer both a macro and micro perspective of the field: from understanding its research trajectory to demonstrating how careful tuning can unlock the potential of underperforming models for specialized summarisation tasks. The findings aim to inform both researchers and practitioners seeking to advance summarisation technologies in theory and application.