STUDY AND DEVELOPMENT OF EFFICIENT DATA COMPRESSION TECHNIQUES

Abstract

The explosive rise of digital data across healthcare, environmental monitoring, industrial automation, and smart cities has intensified the need for efficient and reliable data compression. Resource-constrained platforms such as wireless sensor networks (WSNs) and Internet of Things (IoT) devices face the sharpest challenges: they operate with limited memory, energy, and processing power, yet must transmit critical information without error. Conventional compression methods are not always suited to these constraints, and because the data cannot tolerate distortion, lossless compression is essential. The central aim of this thesis is to study existing techniques and to develop new methods of lossless compression that are faster, lighter, and more adaptive to practical environments. The work unfolds in six major phases, each addressing a distinct gap. Phase 1 begins with a comprehensive benchmarking study of classical algorithms such as Huffman, RLE, LZW, Zstd, LZMA, and LZ4 across synthetic data, real sensor traces, and Indic- language corpora. This baseline analysis revealed clear limitations in speed, memory demand, and energy efficiency when applied to low-power devices. Building on this, Phase 2 introduces a syntactic approach, the LZWP algorithm, which improves throughput by refining dictionary updates while maintaining compression ratio, making it suitable for real-time IoT workloads. In Phase 3, the focus shifts to semantic compression through ontology-driven frameworks. An ontology-based healthcare pipeline and OntoRLE for WSNs demonstrated that lightweight semantic preprocessing could uncover hidden structure in data, leading to better compressibility without compromising accuracy. Phase 4 explores hybrid pipelines, combining algorithms such as cascade of Zstd and LZ4HC, then applying them to Devanagari Hindi corpora and IoT text streams. These cascaded methods achieved a stronger balance between compression ratio, speed, and decompression efficiency compared to standalone techniques. vi Phase 5 presents one of the thesis’s most significant contributions: MOR-ALDC (Memory-Optimised Residual Adaptive Lossless Data Compression). By combining residual transformation with canonical Huffman coding, MOR-ALDC reduced memory requirements and improved both compression efficiency and execution speed. This design was particularly effective for WSNs, where memory footprint and energy usage are critical bottlenecks. The final stage, Phase 6, brings these advances closer to deployment through Artificial Intelligence guided models and hardware acceleration. A decision module was integrated into an FPGA-based compressor to determine, in real time, whether compression should be applied and which algorithm is best suited under given energy and bandwidth conditions. This edge framework, validated through prototype experiments, confirmed that intelligent, adaptive compression can operate reliably at scale in IoT systems while saving energy and improving responsiveness. Across all phases, the proposed methods consistently outperformed classical baselines in compression ratio, memory footprint, throughput, and energy savings. These results carry significant implications for real-world applications. In environmental monitoring, the methods enable long-lived sensor deployments in remote or sensitive areas. In healthcare, they allow reliable, low-cost transmission of patient data in telemedicine settings. In digital inclusion, they improve efficiency for Indic-language text, supporting local-language applications and e-governance. And in IoT and edge computing, the Artificial Intelligence guided framework contributes to greener, more sustainable digital infrastructure. The research also opens clear directions for future work. These include on-node continual learning to adapt decisions as data evolves, dynamic orchestration of hybrid pipelines, and semi-automatic ontology generation to reduce human effort in new domains. Further work on cross-sensor compression, ASIC-level implementations, lightweight security integration, and long-term field trials will strengthen both technical maturity and real-world reliability. Overall, the thesis progresses systematically from benchmarking to syntactic, semantic, hybrid, and statistical approaches, before advancing into Artificial vii Intelligence driven hardware acceleration. Each stage responds to specific gaps, while together they form a coherent framework for efficient, adaptable, and deployable lossless compression. Beyond technical contributions, the research highlights broader social impact by supporting sustainable sensor networks, affordable healthcare, inclusive digital services, and more reliable critical systems.