HAUSDORFF DISTANCE-MIXNET: A HYBRID CONVOLUTIONAL NEURAL NETWORK-TRANSFORMER NETWORK WITH BOUNDARY-AWARE MECHANISMS FOR POLYP SEGMENTATION

Abstract

Colorectal cancer is one of the most preventable cancers. Most colorectal cancers develop from benign polyps that can be removed if found early on colonoscopy. Automated polyp segmentation helps in this regard, but current approaches suffer from a long-standing dilemma that convolutional networks can model fine texture information but fail to model long-range dependency, while transformers can model global structure but ignore fine boundaries. Most hybrid methods simply add these representations in a naive manner and are trained with region-overlap losses like Dice that are insensitive to the worst-case boundary errors of interest in the clinical setting. In this thesis, we propose a hybrid dual branch architecture, namely HD-MixNet, for boundary-aware polyp segmentation. The model consists of a convolutional branch with Res2Net and a Swin Transformer branch, both streams running in parallel. They are combined with a Boundary-Aware Mix-Fusion (BAMF) module, which uses learned edge-gated blending to prevent the boundary suppression problem in attention-based fusion. We explicitly supervise boundaries to YOHO-inspired Edge-Detect and Boundary-Enhance blocks, and train with a composite objective including a differentiable distance-transform approximation of the Hausdorff distance, aligning optimization to worst-case boundary risk, rather than simply overlap . HD-MixNet achieved a mean dice score of 0.896 on Kvasir-SEG benchmark and competitive region-level segmentation with a compact dual-branch design. We also critically analyse the model boundary behaviour and investigate the relation between the overlap and the Hausdorff performance of the model and the factors that influence the worst-case boundary error.