← All publications

1 MIN READ

journal · 2025 · Published · open access (CC BY 4.0)

Attention-Guided Erasing

Can a model decide what not to erase?

Paper
Attention-guided erasing for enhanced transfer learning in breast abnormality classification
Authors
Adarsh Bhandary Panambur · Sheethal Bhat · Hui Yu · Prathmesh Madhu · Siming Bayer · Andreas Maier
Venue
International Journal of Computer Assisted Radiology and Surgery, 20(3), 433–440
Self-Supervised LearningMedical ImagingExplainable AI
FIG 1 / METHOD
(a) DINO pretraining on the target cohort. (b) The six final-layer attention heads become binary masks and erased inputs. (c) Each head is tried during transfer learning; the one with the highest validation F1 is kept. My figure — from the paper and my PhD thesis ↗
05 Classification tasks Whole-image and ROI-level
03 Public cohorts VinDr-Mammo · CDD-CESM · CBIS-DDSM
05× Runs per setting Mean macro F1 ± std reported
4 / 5 Significant gains Mass ROIs were the exception

IN ONE SENTENCE

Summary

Use a self-supervised model's own attention to choose which pixels an augmentation is allowed to destroy — then check the four tasks where that helps and the one where it does not.

01 / THE QUESTION

Random erasing does not know what a lesion is.

Random erasing cuts out an arbitrary rectangle to regularize training. On a mammogram it can just as easily remove the dense tissue or the calcification cluster the classifier is supposed to learn from.

So: could a model that has never seen a label still say which pixels are worth keeping?

02 / THE IDEA

Let self-supervised attention be the erasing policy.

A ViT-S/16 is pretrained with DINO on the target cohort. Its six final-layer attention maps are thresholded into binary masks; everything outside the mask is erased with a chosen probability during fine-tuning.

Which head to trust is task-dependent, so each of the six is tried and the one with the best validation F1 is used — never the test set.

  1. 01Mammogram
  2. 02DINO pretraining
  3. 03Attention heads
  4. 04Binary mask
  5. 05Guided erasing
  6. 06Fine-tuned classifier

03 / WHAT WE TESTED

Five tasks, three cohorts, whole images and small crops.

Two whole-image tasks (BI-RADS breast density in digital mammography, malignancy in contrast-enhanced mammography) and three ROI-level tasks (calcification, CEM lesions, masses). Every setting was run five times against a no-erasing and a random-erasing baseline.

20,000 VinDr-Mammo images 5,000 patients · 4,000-image test set
1,003 CDD-CESM images Cairo University · normal / benign / malignant
326 Calcification test ROIs CBIS-DDSM
378 Mass test ROIs CBIS-DDSM

04 / MAIN RESULT

It works where attention can isolate the evidence.

AGE gave the best mean macro F1 in all five tasks, significantly so in four. On CEM whole images it also cut run-to-run variance about fourfold — the augmentation is acting as a regularizer, not just a lucky seed.

The mass task is the honest counter-example: the lesion already occupies most of the crop, so there is almost no disposable background to erase and the gain vanishes into the noise.

Mean macro F1-score ± standard deviation over 5 runs, as reported in the paper. NE = no erasing, RE = random erasing.
Task (test size)NEREAGEp
Breast density, VinDr-Mammo (4,000)0.55940.56910.5910< 0.0001
Malignancy, CDD-CESM image (116)0.60760.64060.6579< 0.0001
Calcification ROI, CBIS-DDSM (326)0.68030.68460.6892< 0.0001
Lesion ROI, CDD-CESM (116)0.76170.78590.79200.0227
Mass ROI, CBIS-DDSM (378)0.55300.54710.5537n.s.

Standard deviations, per-class scores and the full ablation over erasing probabilities are in the open-access paper. Two-tailed unpaired t-test; n.s. = not significant.

FIG 2 / ATTENTION HEADS
The six attention heads for each task (T1 density, T2 CEM malignancy, T3 calcification ROI, T4 CEM ROI, T5 mass ROI). Red marks the head selected by validation performance. My figure — from the paper and my PhD thesis ↗
FIG 3 / WHAT AGE DOES
Input → selected attention → binary mask → erased image, for two examples of each task. This is what the classifier actually sees during training. My figure — from the paper and my PhD thesis ↗

05 / WHY IT MATTERS

The dataset can supply its own augmentation policy.

No radiologist masks, no extra annotation budget — the augmentation is derived from unlabelled images of the same cohort, which is the setting most medical imaging projects are actually in.

And because the method is reported as conditional rather than universal, it is usable: you can predict in advance whether it will help a given task.

LIMITATION

What this does not show

The attention maps are weak, dataset-derived localizations used to shape augmentation. They are not lesion segmentations, and they should not be read as an explanation of a diagnosis. All results are retrospective.

READ IT YOURSELF

Paper · DOI · Code · Citation

Paper ↗DOI ↗PDF ↗
View original abstract + citation

Abstract

Purpose: We evaluate the effectiveness and generalizability of attention-guided erasing (AGE), a data augmentation technique initially proposed for breast density classification, across various tasks in breast abnormality classification and different mammographic imaging modalities. Methods: A Vision Transformer is pretrained with DINO on the target dataset; self-attention maps of the final-layer attention heads are converted into binary masks that preserve regions of interest while background regions are erased with a random probability during downstream transfer learning. The attention head is treated as a hyperparameter and selected by validation F1-score. Results: AGE achieved the best mean macro F1-score in all five evaluated tasks, with statistically significant improvements in four of them. Conclusion: Self-supervised attention can provide a task-aware augmentation policy without pixel-level annotations, but the benefit depends on whether the relevant structure is localized within the input.

Citation

A. Bhandary Panambur, S. Bhat, H. Yu, P. Madhu, S. Bayer, A. Maier. Attention-guided erasing for enhanced transfer learning in breast abnormality classification. International Journal of Computer Assisted Radiology and Surgery, 20(3):433–440, 2025. doi:10.1007/s11548-024-03317-6

RESEARCH FIGURE / FULL VIEW

Related work

Connected publications

BE-WISE architecture: pre-contrast, post-contrast and subtraction MRI volumes stacked into a (32, 3, 224, 224) tensor, encoded by a shared Swin-Tiny slice encoder into (32, 768) embeddings, then split into a slice-level lesion predictor producing a per-slice probability curve and a breast-level classifier producing normal, benign or malignant

Bildverarbeitung für die Medizin (BVM) 2026, 10–17

BE-WISE: Breast MRI Evaluation with Weakly-Informed Slice-Level Explanation

Published

Breast MRI models usually need lesions outlined voxel by voxel, which does not scale. BE-WISE asks how far one annotated slice per exam gets you. Pre-contrast, post-contrast and subtraction volumes are stacked and encoded slice by slice with a shared Swin-Tiny transformer. One branch predicts a per-slice lesion probability against a Gaussian target centred on the radiologist's slice; the other pools the same embeddings by attention into a normal / benign / malignant decision. On the multicenter ODELIA dataset this reached test AUC 0.8683 and an ODELIA score of 0.7098, against 0.7968 and 0.5695 for a medical slice transformer baseline.

Explainable AIMedical ImagingDetection / Localization
1 min read → Paper ↗
BE-WISER architecture: pre-contrast, post-contrast and subtraction inputs go through a SwinV2-Tiny slice encoder into slice embeddings, which split into a breast-level classifier module with ABMIL attention, feature aggregation, global head and global logits, and a weak slice-level lesion localization branch with a slice head, slice logits and per-slice class probabilities over normal, benign and malignant

Research Square preprint (in review)

BE-WISER: Class-Aware Weak Slice-Level Supervision for Breast MRI

Preprint · in review

BE-WISE could say which slice contained something; BE-WISER says what that something is. Each slice inside the Gaussian region around the radiologist's marked slice inherits the exam's class — benign or malignant — while the rest are labelled normal, so the slice head becomes three-class instead of binary. On the latest multicenter ODELIA release (926 / 230 / 292 examinations from six institutions) this reached breast-level AUC 0.864 and an ODELIA score of 0.699, against 0.807 and 0.591 for a breast-level baseline. Localization also improved: Hit@1 rose from 0.554 to 0.632 and the mean slice distance fell from 2.838 to 2.505 slices.

Explainable AIDetection / LocalizationMedical Imaging
1 min read → Paper ↗
Three-stage architecture: an input mammogram through a SigLIP vision encoder and linear projection into visual tokens, a text instruction through a tokenizer into text token embeddings, both combined into a Gemma 3 4B model with 34 transformer layers generating a report autoregressively, then attention weights extracted per token, aggregated across heads and layers and bicubically interpolated to produce one heatmap per generated token

Manuscript in preparation — no venue yet

Explainable Vision-Language Models for Mammography Report Generation

Manuscript in preparation · not peer reviewed

This is the follow-up to MammoBLIP, and it asks a different question: not how good the report is, but whether you can see where each word came from. Domain-adapted SigLIP vision encoders are paired with 4B-parameter LLMs (Gemma 3 and MedGemma), and attention is aggregated across heads and layers to produce one heatmap per generated token. The best of six configurations reached ROUGE-1 0.877 and ROUGE-2 0.843, up 0.127 and 0.143 on MammoBLIP. The more interesting finding is negative: attention often lands on the right microcalcification cluster while the diagnosis is still wrong.

Vision-Language ModelsLLM / GenAIExplainable AI
1 min read →