DETECTION O FAI-GENERATED TEXT USING TRANSFORMER MODELS AND STYLOMETRIC FEATURES

Abstract

Large language models (LLMs) — ChatGPT, GPT-4, Gemini, and their successors — have made AI-generated text startlingly difficult to distinguish from content written by people. The advantages are real: quick writing, wide information accessibility, and effective educational aids. However, this power can be used to commit academic dishonesty, spread targeted misinformation, or just lead to a lack of faith in written content overall. Current detectors either rely on shallow vocabulary statistics that split across various domains, or they run more complex deep-learning pipelines that have little qualitative insight into why the text has that AI-generated feel. This thesis fills that void. We present and test three complementary detection systems: (i) a TF-IDF logistic-regression system, (ii) a hybrid (TF-IDF combined with hand crafted stylometric features) fed into a Random Forest Classifier, and (iii) a fine-tuned DistilBERT transformer. One of the defining features of the work is the development of the Sentence Structure Variance Feature (SSVF), a single scalar that captures the overall rhythmic irregularity of the writer’s sentence lengths, typically not found in the output produced by most LLM systems. All experiments use a stratified subset of the Kaggle AI vs. Human Text corpus, com prising 20,000 samples drawn from 487,235 total, with 62.75% human and 37.25% AI-generated content. Stylometric features are extracted from the raw text after a prepro cessing pipeline including lowercasing, URL detection, normalisation of punctuation, cleaning of whitespace, stop-word removal, tokenisation, and lemmatisation. The TF-IDF baseline achieves 97.85% accuracy and 97.12% F1. The hybrid model improves precision to 99.10%, lowering false accusations by approximately 16 fewer false positives per 4,000 documents — a gain of practical value. The best all-round performance comes from fine-tuned DistilBERT, which achieves 98.75% accuracy and 98.35% F1 score, mainly driven by a recall of 98.68%. Ablation experiments validate the independent contributions of TF-IDF scores, traditional stylometrics, and the SSVF to the overall classification scores.