A team from MiLM Plus, Xiaomi Inc. has released PROVE (Perceptual RemOVal cohErence), an evaluation harness accepted at ACM MM 2026, designed to address limitations of existing metrics for object removal models. Current metrics like PSNR, SSIM, LPIPS, ReMOVE, and CFD frequently rank the outputs of advanced diffusion-based erasers incorrectly, despite their ability to convincingly reconstruct complex scene elements. This discrepancy stems from the inherent ill-posed, one-to-many nature of object erasure, where multiple plausible restorations exist for a single removed region, precluding a unique ground truth. PROVE addresses this by employing two perception-aligned metrics, RC-S for spatial coherence and RC-T for temporal consistency, paired with PROVE-Bench, a two-tier real-world video benchmark. Both metrics operate by performing local distribution matching in a deep feature space, specifically using DINOv2 features, and neither requires a reference video.
The architectural design of PROVE's metrics focuses on local feature analysis. RC-S evaluates spatial coherence by splitting the mask into connected components, expanding their bounding boxes by one third of their side length, and feeding crops to DINOv2. It then computes squared Maximum Mean Discrepancy (MMD) with a Gaussian RBF kernel within a sliding window across the feature map, comparing masked features to local background features. RC-T assesses temporal consistency by jointly cropping adjacent frames under the union of their masks, computing MMD exclusively within the intersection region restored in both frames. PROVE is released as an Apache 2.0 PyTorch repository, functioning as an evaluation harness.
- Technical Requirements:
- Python 3.10+
- PyTorch 2.6+
- Transformers 4.51+
- DINOv2-giant weights
- Mandatory masks (white pixels denote removed object)
Empirical results demonstrate PROVE's alignment with human judgment, outperforming existing metrics. RC-S achieved an average Kendall's τ of 0.59 and Spearman's ρ of 0.66 against human rankings, outperforming ReMOVE (0.26/0.29) and CFD (0.16/0.18), and ranked first on five of six benchmarks. On RORD-Val, RC-S preferred clean images over blurred or region-swapped variants in 100% of cases, compared to 60.06% for ReMOVE and 49.27% for CFD under blur. Ablation studies demonstrated DINOv2's performance advantages (0.59 avg τ) over DINOv3 (0.51) and SAM (0.44); dropping the sliding window cost 0.11 τ, and swapping MMD for cosine similarity cost 0.07 τ. RC-S also demonstrated efficiency, operating 13.7× faster than CFD.