DISENTANGLING RELEVANCE AND SALIENCE: A DUAL SUPERVISION FRAMEWORK FOR VIDEO MOMENT RETRIEVAL

Abstract

Finding specific moments in videos based on natural language descriptions is a core challenge in video understanding. This problem, known as video moment retrieval, requires a system to identify the exact start and end times in an untrimmed video that correspond to a given text query. Current state-of-the-art methods such as SG-DETR use DETR-style architectures where multiple decoder queries each predict a temporal span in a single forward pass. These models are trained with bipartite matching: the Hungarian algorithm assigns each ground-truth moment to exactly one predicted span, and only that matched prediction receives the span regression loss (L1 + GIoU). The remaining predictions are only pushed toward a “no-object” classification and never receive direct localization feedback. As a result, the model does not learn to compare or rank its own predictions against each other. In this thesis, we introduce DualRS, a reinforcement learning framework that pro vides a learning signal to all predictions simultaneously. We apply Group Relative Policy Optimization (GRPO), originally developed for language model alignment, to the set of proposals from a single forward pass. Each proposal is scored using a re ward function that measures its quality (semantic relevance and saliency overlap), and the model is updated to increase the likelihood of high-reward predictions while de creasing that of low-reward ones. Because the decoder already outputs multiple spans deterministically, no repeated sampling or stochastic rollouts are needed because the group structure is inherent to the architecture. We further introduce a dual reward function that combines two complementary signals: semantic relevance and temporal saliency. The semantic reward uses Natural Language Inference (NLI) to measure how well the content within a predicted time span logically supports the query text, computed using DeBERTa-v3 on auto-generated speech transcripts. The saliency reward leverages crowd-sourced human annotations indicating which clips are visually important. Together, these two signals provide a richer training objective than either alone: relevance guides what to retrieve while saliency guides how engaging the retrieved content should be. We also introduce cross-policy exploration, where a frozen copy of the original model contributes additional proposals to the group. This maintains exploration di versity as training progresses and prevents the model’s proposals from collapsing into similar temporal regions. Experiments on the QVHighlights benchmark demonstrate that DualRS achieves 73.42% R1@0.5 on the validation split, surpassing the current state-of-the-art SG DETR(+0.62%R1@0.5). Ontheheld-outtestsplit, DualRSachieves75.15%R1@0.5, outperforming SG-DETRby+0.95%. Theseimprovementsgeneralizetootherdatasets including Charades-STA (71.83%R1@0.5, +1.63%overSG-DETR)andTACoS(31.14% R1@0.5, +1.24% over SG-DETR).