Can attention replace a segmentation network?

The standard density pipeline trains one network to remove background and pectoralis muscle, then a second to classify. That first network needs radiologist annotations or hand-tuned image processing, which is exactly what a new cohort will not have.

The question was whether a self-supervised model already knows enough about where breast tissue is to skip that step.

A DeiT-small ViT is pretrained with DINO on the VinDr-Mammo training split — no labels. Of its six final-layer attention heads, the one whose activation covers only a small number of pixels turns out to track dense tissue; it is thresholded into a binary mask.

During fine-tuning, everything outside that mask is erased with probability P. The classifier is pushed towards the tissue and away from the background, without anyone drawing a contour.

VinDr-Mammo: 20,000 full-field digital mammograms from 5,000 patients at two Hanoi hospitals, BI-RADS density annotated by consensus of three radiologists. Density C dominates with 15,292 of the 20,000 images; density A has just 100.

AGE was compared against no erasing and against random erasing at four probabilities, five runs each, on the original 4,000-image test split.

AGE reached mean macro F1 0.5910 at P=0.6, ahead of no erasing (0.5594), random erasing (0.5691) and the published ResNet-34 and EfficientNet-B2 single-view baselines (0.504 and 0.5525).

The mechanism shows in the probability sweep: random erasing decays as you erase more, because it starts removing tissue. AGE stays above the baseline throughout — it only ever removes background.

Removing the annotated segmentation stage is what makes density assessment portable to a cohort that has labels but no masks.

This paper is the origin of the attention-guided erasing line: the journal follow-up tested it across five tasks and three modalities, and found both where it generalizes and where it stops.