← All publications

1 MIN READ

preprint · 2026 · Manuscript in preparation · not peer reviewed

Explainable mammography VLM

Can you see where each word of the report came from?

Not peer reviewed — treat the numbers below as preliminary.

Paper
Explainable Vision-Language Models for Mammography Report Generation
Authors
Adarsh Bhandary Panambur · Karim Mohamed Sakr · Tri-Thien Nguyen · Andreas Maier · Siming Bayer
Venue
Manuscript in preparation — no venue yet
Vision-Language ModelsLLM / GenAIExplainable AI
FIG 1 / ARCHITECTURE
Stage I encodes the mammogram with a domain-adapted SigLIP tower. Stage II generates the report with a 4B LLM. Stage III turns the attention behind each generated token into a heatmap over the input image. My figure — from the manuscript
06 Configurations trained 2 LLMs × 3 vision encoders
0.877 ROUGE-1 MammoBLIP: 0.750
+17% Relative ROUGE-1 gain +0.127 absolute
1 : 1 Heatmap per token Every generated word is grounded

IN ONE SENTENCE

Summary

Map every generated report token back to a region of the mammogram — then find out that the model can point at the right lesion and still reach the wrong conclusion.

01 / THE QUESTION

A fluent report is not an accountable one.

MammoBLIP could write a structured mammography report. It could not tell you which part of the image produced the phrase "BI-RADS 5" — and in screening, that is the part a radiologist needs in order to disagree with it.

So the question became: can the report be grounded token by token, and does grounding actually track correctness?

02 / THE IDEA

Domain-adapted SigLIP, an LLM, and one heatmap per token.

Three vision encoders — a base tower, Mammo-SigLIP and Mammo-MedSigLIP — are each projected into two 4B-parameter language models, Gemma 3 and the medically specialized MedGemma. Six configurations in total.

During generation, attention for each output token is aggregated across heads and layers and interpolated back to the input resolution. Every word in the report gets its own map.

  1. 01Mammogram
  2. 02Domain-adapted SigLIP
  3. 03Projection
  4. 044B LLM (Gemma 3 / MedGemma)
  5. 05Generated report
  6. 06Per-token attention map

03 / WHAT WE TESTED

The same corpus as MammoBLIP, so the comparison is fair.

Over 81,076 full-field digital mammograms from the same five public multi-institutional sources, with structured annotations for view, laterality, abnormality type, pathology, BI-RADS category and breast density converted into reports and multi-turn question-answer pairs.

Reporting quality was scored with ROUGE-1, ROUGE-2, ROUGE-L and exact match; grounding was assessed qualitatively through per-token attention case studies.

81,076 Mammograms Same public corpus as MammoBLIP
4B LLM parameters Gemma 3 and MedGemma
03 Vision encoders Base · Mammo-SigLIP · Mammo-MedSigLIP
0.524 Exact match Best configuration

04 / MAIN RESULT

Better text — and a warning about reading too much into attention.

The best configuration reached ROUGE-1 0.877 and ROUGE-2 0.843, up 0.127 and 0.143 on MammoBLIP, which supports the case for adapting the vision encoder to mammography rather than using a general-purpose tower.

The grounding maps behave sensibly: diagnostic and recommendation tokens converge on the microcalcification cluster while anatomical tokens show a view-specific bias. But correct localization and correct diagnosis came apart — a model can look in the right place and still be wrong.

Report-generation metrics, best configuration versus the MammoBLIP baseline.
MetricMammoBLIPThis workΔ
ROUGE-10.7500.877+0.127
ROUGE-20.7000.843+0.143
ROUGE-L0.7400.874+0.134
Exact match—0.524—

MammoBLIP values are its reported overall test scores on 13,242 report pairs. The manuscript does not state which of the six configurations achieved the best result.

FIG 2 / TOKEN-LEVEL ATTENTION
One heatmap per generated token for a single case. The diagnostic and recommendation tokens — calcification, BI-RADS 5, malignancy, urgent, biopsy — converge on the same microcalcification cluster. My figure — from the manuscript

05 / WHY IT MATTERS

The negative result is the contribution.

Per-token grounding gives a radiologist a way to audit a generated report sentence by sentence instead of trusting it as a block.

And the finding that localization and diagnosis can diverge sets a limit on what attention maps are evidence for. Calibration and prospective validation have to come before any deployment claim.

LIMITATION

What this does not show

Not peer reviewed. Evaluation is lexical overlap against label-derived reports, with no reader study and no clinical validation, and the manuscript does not say which configuration won.

READ IT YOURSELF

Paper · DOI · Code · Citation

View original abstract + citation

Abstract

Vision-language models are changing medical image understanding, but clinical translation in mammography is constrained by high-resolution grayscale imaging, subtle lesions, breast-density masking, and the need for interpretable evidence in a high-stakes screening setting. This work pairs domain-adapted SigLIP vision encoders with 4-billion-parameter language models and adds token-level attention visualization, so each generated report token can be mapped back to a spatial region of the mammogram. Six configurations were trained: two LLM backbones (Gemma 3 and the medically specialized MedGemma) crossed with three vision encoders (base vision tower, Mammo-SigLIP, Mammo-MedSigLIP), over the same 81,076-image public multi-institutional corpus used for MammoBLIP.

Citation

A. Bhandary Panambur, K. M. Sakr, T.-T. Nguyen, A. Maier, S. Bayer. Explainable Vision–Language Models for Mammography Report Generation. Manuscript in preparation, 2026.

RESEARCH FIGURE / FULL VIEW

Related work

Connected publications

BE-WISE architecture: pre-contrast, post-contrast and subtraction MRI volumes stacked into a (32, 3, 224, 224) tensor, encoded by a shared Swin-Tiny slice encoder into (32, 768) embeddings, then split into a slice-level lesion predictor producing a per-slice probability curve and a breast-level classifier producing normal, benign or malignant

Bildverarbeitung für die Medizin (BVM) 2026, 10–17

BE-WISE: Breast MRI Evaluation with Weakly-Informed Slice-Level Explanation

Published

Breast MRI models usually need lesions outlined voxel by voxel, which does not scale. BE-WISE asks how far one annotated slice per exam gets you. Pre-contrast, post-contrast and subtraction volumes are stacked and encoded slice by slice with a shared Swin-Tiny transformer. One branch predicts a per-slice lesion probability against a Gaussian target centred on the radiologist's slice; the other pools the same embeddings by attention into a normal / benign / malignant decision. On the multicenter ODELIA dataset this reached test AUC 0.8683 and an ODELIA score of 0.7098, against 0.7968 and 0.5695 for a medical slice transformer baseline.

Explainable AIMedical ImagingDetection / Localization
1 min read → Paper ↗
BE-WISER architecture: pre-contrast, post-contrast and subtraction inputs go through a SwinV2-Tiny slice encoder into slice embeddings, which split into a breast-level classifier module with ABMIL attention, feature aggregation, global head and global logits, and a weak slice-level lesion localization branch with a slice head, slice logits and per-slice class probabilities over normal, benign and malignant

Research Square preprint (in review)

BE-WISER: Class-Aware Weak Slice-Level Supervision for Breast MRI

Preprint · in review

BE-WISE could say which slice contained something; BE-WISER says what that something is. Each slice inside the Gaussian region around the radiologist's marked slice inherits the exam's class — benign or malignant — while the rest are labelled normal, so the slice head becomes three-class instead of binary. On the latest multicenter ODELIA release (926 / 230 / 292 examinations from six institutions) this reached breast-level AUC 0.864 and an ODELIA score of 0.699, against 0.807 and 0.591 for a breast-level baseline. Localization also improved: Hit@1 rose from 0.554 to 0.632 and the mean slice distance fell from 2.838 to 2.505 slices.

Explainable AIDetection / LocalizationMedical Imaging
1 min read → Paper ↗
conference 2026 Vision-Language Models · Foundation Models

European Congress of Radiology (ECR) 2026, scientific exhibit C-15780

Lesion-Aware AI for Mammography: Multi-Dataset Pretraining with ROI-Guided Contrastive Learning and Clinical Image Retrieval

Accepted

Global contrastive pretraining lets a mammography model align on breast density or view rather than on the lesion, because those cues are enough to match an image to its report. This work adds region-of-interest guidance to multi-dataset contrastive pretraining, so localized pathology has to contribute to the representation, and then tests the embeddings on clinical image retrieval — finding visually and diagnostically similar prior cases. Accepted as a scientific exhibit at ECR 2026; quantitative results are not yet publicly available.

Vision-Language ModelsFoundation ModelsMedical Imaging
Details → DOI ↗