Can a mammogram be turned straight into a report?

Paper: MammoBLIP: End-to-End Mammography Report Generation with Vision-Language Models and Public Multi-Institutional Datasets
Authors: Adarsh Bhandary Panambur · Sebastian Wind · Siming Bayer · Andreas Maier
Venue: 2025 IEEE Nuclear Science Symposium, Medical Imaging Conference and Room Temperature Semiconductor Detector Conference (NSS/MIC/RTSD), Yokohama

Vision-Language ModelsLLM / GenAIMedical Imaging

MammoBLIP pipeline: five public dataset sources feed a standardized preprocessing step; a mammogram goes through a frozen EVA-CLIP image encoder and the paired report through a tokenizer, both into the MammoBLIP transformer, whose projected embeddings drive a frozen BioMedLM to produce a generated radiology report — Five public repositories are standardized into one corpus. Only the MammoBLIP transformer and the projection heads are trained; EVA-CLIP and BioMedLM stay frozen (marked f). *My figure — from the paper and my PhD thesis*

81,076 Mammograms curated VinDr · RSNA · CMMD · INbreast · KAU

13,242 Test report pairs Held out across all five sources

65.36 BLEU overall

7.04 Flesch–Kincaid grade Readability of generated reports

IN ONE SENTENCE

Freeze a vision foundation model, freeze a biomedical LLM, train only the small bridge between them — and see how close the generated report gets to the reference.

01 / THE QUESTION

A BI-RADS score is not a report.

Breast imaging models are built one task at a time — detect a mass, classify malignancy, estimate risk. A radiologist instead writes a paragraph that ties laterality, view, findings, BI-RADS category and a recommendation together.

Whether an image-to-text model can produce that paragraph for mammography had not been examined at scale.

02 / THE IDEA

Train the bridge, not the backbones.

A frozen EVA-CLIP Vision Transformer supplies patch features; a frozen BioMedLM writes the text. Between them sits a small trainable transformer that projects image and text into one 256-dimensional space, aligned with a symmetric contrastive loss.

The image embedding is inserted between the prompt and the target tokens, and the language loss is computed only over the report. Everything fits on one GPU.

01Mammogram
02Frozen EVA-CLIP
03Bridge transformer
04Shared 256-D space
05Frozen BioMedLM
06Structured report

03 / WHAT WE TESTED

One corpus out of five public repositories.

VinDr-Mammo, RSNA, CMMD, INbreast and KAU-BCMD were put through a single DICOM-to-PNG, cropping and normalization pipeline, and their structured labels were converted into consistent radiology-style reports.

Stratified sampling kept the clinical-attribute distributions comparable across splits.

59,089 Train images

8,745 Validation images

13,242 Test images

05 Public sources All openly licensed

04 / MAIN RESULT

Fluent and semantically close — and dataset-dependent.

Over all 13,242 test pairs the generated reports reached BLEU 65.36, ROUGE-1 0.75, BERT-F1 0.88 and SBERT similarity 0.91, at a reading level around grade seven.

The spread between sources is the more interesting number: standardized, homogeneous reporting (KAU) is far easier to reproduce than an older, more heterogeneous cohort (INbreast).

Reported generation metrics by test dataset (test-set size in brackets).
Dataset	BLEU	ROUGE-1	BERT-F1	SBERT
KAU-BCMD (334)	85.86	0.92	0.96	0.93
VinDr-Mammo (4,000)	75.45	0.84	0.92	0.92
RSNA (8,285)	60.66	0.71	0.86	0.91
CMMD (560)	56.12	0.71	0.86	0.90
INbreast (63)	53.01	0.70	0.86	0.93
Overall (13,242)	65.36	0.75	0.88	0.91

Values exactly as reported. No internal baseline or ablation was run, so these are absolute scores rather than a comparison.

05 / WHY IT MATTERS

Text output is checkable output.

A generated report can be read and corrected line by line, which a single probability cannot. That makes it a more useful object for clinical scrutiny — and a better starting point for grounding and explainability work.

This model became the baseline for the follow-up work on token-level attention grounding.

LIMITATION

What this does not show

There is no baseline model, no ablation and no reader study, and the references are reports derived from dataset labels rather than free-text clinical dictation. High overlap scores therefore say the model learned the label-to-report mapping well, not that it is clinically reliable.

READ IT YOURSELF

Paper · DOI · Code · Citation

DOI ↗

View original abstract + citation

Abstract

Breast imaging AI is largely task-specific: lesion detection, risk stratification or malignancy classification, each in isolation. None of these produce the structured, comprehensive report a radiologist writes. MammoBLIP is an end-to-end vision-language framework that generates mammography reports directly from an image. A frozen EVA-CLIP Vision Transformer supplies patch features, a lightweight MedBLIP-style transformer projects image and text into a shared 256-dimensional space with a symmetric contrastive loss, and a frozen BioMedLM decoder generates the report autoregressively. The model is trained on 81,076 mammograms curated from five public repositories with a unified preprocessing and label-to-report pipeline, and evaluated on 13,242 held-out report pairs.

Citation

A. Bhandary Panambur, S. Wind, S. Bayer, A. Maier. MammoBLIP: End-to-End Mammography Report Generation with Vision-Language Models and Public Multi-Institutional Datasets. 2025 IEEE NSS/MIC/RTSD, Yokohama, Japan, 2025. doi:10.1109/NSS/MIC/RTSD57106.2025.11286311