COMPARATIVE EVALUATION OF TRANSFORMER-BASED GENERATIVE MODELS FOR ABSTRACTIVE TEXT SUMMARIZATION: A SYSTEMATIC REVIEW AND IMPLEMENTATION STUDY WITH FACTUAL CONSISTENCY ANALYSIS

Abstract

The rapid growth in the amount of digital textual information available on news websites, academic databases, social media sites, legal databases corporate information systems has put a significant cognitive burden on human readers who seek to glean useful informa tion from the information available. Abstractive summarization, which entails the need for computers to construct coherent summaries using new sentences and not snippets of sentences taken directly from the documents, is one of the most difficult and crucial prob lems in the area of natural language processing. The emergence of large-scale pre-trained transformers has revolutionized abstractive summarization and T5, BART, PEGASUS and FLAN-T5 have set successively better benchmarks on standard evaluation datasets. However, abstractive summarization is still riddled with many unresolved problems, in cluding the problem of hallucination where generative models generate information that is not present in the source document, the inability of ROUGE metrics to evaluate se mantic accuracy, lack of statistical rigor in performance comparison assertions and heavy computational costs associated with summarization. The thesis aims to propose a dual contribution study which will try to fill this gap. In two methodologically connected parts. The first step is a Systematic Literature Re view (SLR) is carried out using methodology that was introduced by The report was compiled by Kitchenham and Charters, which is a collection of 42 scholarly papers pub lished between 2020 and 2025, that documents how generative AI methods have evolved for text.2025, that chronicles the evolution of generative AI methods for text. sum marization, highlights the state of the art of the model architecture and Identifies the important constraints and future outlook of the benchmark datasets used perspectives. Second, a comparison analysis of how second language English is implemented was con ducted. We use 4 popular transformer models: T5-Base (220M), BART-Large (400M), The PEGASUS-Large (570M) and FLAN-T5-Large (780M) is performed with the follow ing cases: experimental setup with two widely used news benchmarks, strictly unified. (CNN/DailyMail and XSum) and the out-of-domain dialogue dataset SAMSum in the zero-shot learning setup. across three metrics: surface-level lexical quality estimated using ROUGE-1/2/L, semantic similarity estimated using BERTScore and factual consistency estimated using AlignScore. The empirical results reveal that PEGASUS-Large obtains the highest ROUGE-1 score of 47.21 on the highly compressed XSum dataset due to its task-oriented pre-training objective of Gap Sentence Generation, while BART-Large and PEGASUS-Large obtain nearly identical ROUGE-1 scores on CNN/DailyMail (44.16 vs. 44.17) without statis tically significant differences. Importantly, an empirically validated and novel result in this work pertains to the factual consistency aspect: FLAN-T5-Large obtains the highest AlignScore on both datasets (CNN/DailyMail: 0.512; XSum: 0.461) even though it does not obtain the highest ROUGE scores, clearly demonstrating empirically that instruction tuning is more beneficial for factual consistency than lexical overlap. In addition, FLAN T5-Large exhibits substantially better cross-domain generalizability on zero-shot evalua tion on SAMSum in terms of ROUGE-1 (31.56 vs. 24.37) than BART-Large, thus making instruction tuning the preferable technique for practitioners who need domain-agnostic summarization models. T5-Base, which takes only 0.27 seconds per sample to make an inference and requires 1.9 GB of GPU memory, presents the best cost-performance trade off