SEMANTIC REGION-AWARE FACIAL ATTRIBUTE EDITING VIA SEGFORMER PARSING AND DIFFUSION INPAINTING

Abstract

Facial attribute editing, which involves making modifications to a specific area of the face without changing other areas, is still a difficult task due to its inherent nature of being on the cusp of semantic segmentation and generative image synthesis. The existing GAN-based methods, like StarGAN and AttGAN, have issues with controlling the edit's position, while the diffusion-based techniques, which do not employ masks, do not restrict the edit to the proper area. In this work, I introduce SemFaceDiff, which incorporates both capabilities through a transformer-based face parsing followed by latent diffusion-based inpainting in a three-step pipeline. Then, the segmentation mask is improved using morphological binary dilation and Gaussian boundary feathering (σ=7). The actual inpainting is carried out with Stable Diffusion XL Inpainting, which utilizes a pair of CLIP-based prompt encoders. The presented solution allows for editing 11 facial attributes in an iterative fashion and produces photo-realistic results, measured by the high SSIM score, low LPIPS score, and CLIP Faithfulness score.