PARAKH:ACOMPREHENSIVE FRAMEWORKFOREVALUATINGSOCIAL BIAS IN HINDI-LANGUAGE LARGE LANGUAGEMODELS

Abstract

Large Language Models (LLMs) are increasingly deployed across India, yet infras tructure for evaluating their social biases in Indian languages remains absent. Exist ing benchmarks (BBQ, CrowS-Pairs, WinoBias, BOLD) are English-centric and miss India-specific bias axes such as caste discrimination, religious communalism, and re gional prejudice. This thesis presents PARAKH(ProbingAIResponsesAgainstKnownHindustani-societal biases), the first comprehensive Hindi-language LLM bias benchmark. PARAKH com prises 1,000 expert-crafted Hindi prompts spanning eight bias categories (Caste, Reli gious, Gender, Regional & Linguistic, Colorism, Class & Economic, LGBTQ+, Age &Disability), four difficulty levels, and five prompt types. Five LLMs are evaluated — Llama 3.18B,Qwen38B,Gemma29B,Gemini2.5Flash-Lite,andSarvam-12B—us ing a novel five-dimensional composite scoring rubric (Harm, Stereotype, Sycophancy, Refusal Quality, Counterfactual Fairness) with automated dual-judge validation. Evaluation of 1,048 judgments reveals significant inter-model variation. Gemma 2 9B performs best (mean composite 1.55, 76.8% proper refusal rate), while Qwen3 8B per forms worst (mean3.10, 30%failed-refusal rate). Sarvam-1 2B, despite only 2B param eters, matches Llama 3.1 8B (2.26 vs. 2.23), suggesting India-focused training partially compensates for size. Gender Bias is the hardest category for 3 of 5 models, and Role Play prompts most effectively bypass safety mechanisms (mean 2.92 vs. 1.57 for Opin ion Seeking). Inter-judge agreement (κ = 0.384) is consistent with human annotator levels in bias literature. Notably, one modelproducedanarrative justifying a Dalit engineer’s dismissal because “एकनीचीजातके कोऊंचीजातके ठेके दारोंको नदशदेनेकाअधकार नहीं” (a lower-caste person has no right to give orders to upper-caste contractors) — with no refusal mechanism activating. PARAKH establishes the first reproducible infras tructure for Hindi-language LLM bias evaluation.