← All publications

preprint · 2025 · Under internal review · no results disclosed

Towards Foundational Models in Mammography: Leveraging Vision-Language Models for Radiological Reasoning

Adarsh Bhandary Panambur · Sheethal Bhat · Tri-Thien Nguyen · Siming Bayer · Andreas Maier

Not peer reviewed.

01 / SUMMARY

What this paper does

A journal manuscript on vision-language foundation models for mammography, focused on whether image-text representation learning can support structured radiological reasoning rather than a single classification output. It is listed here for completeness of the record: the work is under internal review, no venue has been assigned, and no quantitative results are publicly available, so this page deliberately contains no numbers. The related and more developed strand of this line of work — token-level attention grounding of generated mammography reports — is written up separately on this site.

02 / METHOD

How

Vision-language representation learning connecting mammography images with radiology language and reasoning-oriented model outputs. Methodological detail is withheld while the manuscript is under review.

03 / FINDINGS

Reported results

  • Under internal review. No quantitative results are publicly disclosed and none are reproduced here.
04 / WHY IT MATTERS

Relevance

Reasoning-oriented foundation models could make mammography AI more structured and auditable than a single-score classifier, which is the direction the rest of this work also points in.

05 / LIMITATION

What it does not show

Not peer reviewed, no venue, and no public results. Listed for completeness only.

06 / REFERENCE

Citation

A. Bhandary Panambur, S. Bhat, T.-T. Nguyen, S. Bayer, A. Maier. Towards Foundational Models in Mammography: Leveraging Vision-Language Models for Radiological Reasoning. Journal submission, under internal review, 2025.

View original abstract

A journal submission on vision-language foundation models for mammography, focused on image-text representation learning and structured radiological reasoning rather than single-task image classification. The manuscript is under internal review and no quantitative results are publicly available.

Related work

Connected publications

Three-stage architecture: an input mammogram through a SigLIP vision encoder and linear projection into visual tokens, a text instruction through a tokenizer into text token embeddings, both combined into a Gemma 3 4B model with 34 transformer layers generating a report autoregressively, then attention weights extracted per token, aggregated across heads and layers and bicubically interpolated to produce one heatmap per generated token

Manuscript in preparation — no venue yet

Explainable Vision-Language Models for Mammography Report Generation

Manuscript in preparation · not peer reviewed

This is the follow-up to MammoBLIP, and it asks a different question: not how good the report is, but whether you can see where each word came from. Domain-adapted SigLIP vision encoders are paired with 4B-parameter LLMs (Gemma 3 and MedGemma), and attention is aggregated across heads and layers to produce one heatmap per generated token. The best of six configurations reached ROUGE-1 0.877 and ROUGE-2 0.843, up 0.127 and 0.143 on MammoBLIP. The more interesting finding is negative: attention often lands on the right microcalcification cluster while the diagnosis is still wrong.

Vision-Language ModelsLLM / GenAIExplainable AI
1 min read →
conference 2026 Vision-Language Models · Foundation Models

European Congress of Radiology (ECR) 2026, scientific exhibit C-15780

Lesion-Aware AI for Mammography: Multi-Dataset Pretraining with ROI-Guided Contrastive Learning and Clinical Image Retrieval

Accepted

Global contrastive pretraining lets a mammography model align on breast density or view rather than on the lesion, because those cues are enough to match an image to its report. This work adds region-of-interest guidance to multi-dataset contrastive pretraining, so localized pathology has to contribute to the representation, and then tests the embeddings on clinical image retrieval — finding visually and diagnostically similar prior cases. Accepted as a scientific exhibit at ECR 2026; quantitative results are not yet publicly available.

Vision-Language ModelsFoundation ModelsMedical Imaging
Details → DOI ↗
journal 2026 Vision-Language Models · Explainable AI

Scientific Reports, 16, article 14688

PatchCLIP enables region specific contrastive health record and image joint training with patch embedding loss

Published · open access (CC BY 4.0)

Contrastive image-text models in radiology align a whole image with a whole report, so their explanations have to be reverse-engineered with saliency methods afterwards. PatchCLIP instead trains a patch-level contrastive objective, so localization is part of the model rather than a post-hoc interpretation. At inference it emits patch-wise prediction maps as well as a global score. Across eight chest X-ray abnormality detection tasks the paper reports the best performance of the methods it compares against, and substantially fewer false positives at comparable sensitivity than saliency-map baselines. Code is released publicly by Siemens Healthineers. I contributed as a co-author.

Vision-Language ModelsExplainable AIDetection / Localization
Details → Paper ↗