← All publications

journal · 2026 · Published · open access (CC BY 4.0)

PatchCLIP enables region specific contrastive health record and image joint training with patch embedding loss

Sheethal Bhat · Awais Mansoor · Bogdan Georgescu · Mathias Zinnen · Pranjal Sahu · Adarsh B. Panambur · Florin C. Ghesu · Sasa Grbic · Andreas Maier

Paper ↗DOI ↗Code ↗
01 / SUMMARY

What this paper does

Contrastive image-text models in radiology align a whole image with a whole report, so their explanations have to be reverse-engineered with saliency methods afterwards. PatchCLIP instead trains a patch-level contrastive objective, so localization is part of the model rather than a post-hoc interpretation. At inference it emits patch-wise prediction maps as well as a global score. Across eight chest X-ray abnormality detection tasks the paper reports the best performance of the methods it compares against, and substantially fewer false positives at comparable sensitivity than saliency-map baselines. Code is released publicly by Siemens Healthineers. I contributed as a co-author.

02 / METHOD

How

Image-patch embeddings are contrasted against text embeddings during joint image-report training, producing patch-wise prediction maps at inference in addition to the global classification score.

03 / FINDINGS

Reported results

  • Reports the strongest performance among the compared methods across eight chest X-ray abnormality detection tasks.
  • Patch prediction maps substantially reduced false positives at comparable sensitivity compared with saliency-based localization.
  • Public code release: github.com/Siemens-Healthineers/patch-clip.
04 / WHY IT MATTERS

Relevance

If a medical image-text model is going to be used as evidence, the region it points at has to come from the model's own objective and not from a saliency approximation applied afterwards.

05 / LIMITATION

What it does not show

Evaluation is on chest radiography abnormality detection benchmarks; transfer to other modalities is not established in this paper.

06 / REFERENCE

Citation

S. Bhat, A. Mansoor, B. Georgescu, M. Zinnen, P. Sahu, A. B. Panambur, F. C. Ghesu, S. Grbic, A. Maier. PatchCLIP enables region specific contrastive health record and image joint training with patch embedding loss. Scientific Reports, 16:14688, 2026. doi:10.1038/s41598-026-52235-x

View original abstract

CLIP-style medical image-text models align a whole image with a whole report, which is enough to classify but not to localize. PatchCLIP adds a patch embedding loss that contrasts image-patch-level embeddings against text embeddings, so that at inference the model produces patch-wise prediction maps alongside the global score. Evaluated across eight chest X-ray abnormality detection tasks, the approach reports state-of-the-art performance and substantially fewer false positives at comparable sensitivity than saliency-based localization.

Related work

Connected publications

BE-WISE architecture: pre-contrast, post-contrast and subtraction MRI volumes stacked into a (32, 3, 224, 224) tensor, encoded by a shared Swin-Tiny slice encoder into (32, 768) embeddings, then split into a slice-level lesion predictor producing a per-slice probability curve and a breast-level classifier producing normal, benign or malignant

Bildverarbeitung für die Medizin (BVM) 2026, 10–17

BE-WISE: Breast MRI Evaluation with Weakly-Informed Slice-Level Explanation

Published

Breast MRI models usually need lesions outlined voxel by voxel, which does not scale. BE-WISE asks how far one annotated slice per exam gets you. Pre-contrast, post-contrast and subtraction volumes are stacked and encoded slice by slice with a shared Swin-Tiny transformer. One branch predicts a per-slice lesion probability against a Gaussian target centred on the radiologist's slice; the other pools the same embeddings by attention into a normal / benign / malignant decision. On the multicenter ODELIA dataset this reached test AUC 0.8683 and an ODELIA score of 0.7098, against 0.7968 and 0.5695 for a medical slice transformer baseline.

Explainable AIMedical ImagingDetection / Localization
1 min read → Paper ↗
BE-WISER architecture: pre-contrast, post-contrast and subtraction inputs go through a SwinV2-Tiny slice encoder into slice embeddings, which split into a breast-level classifier module with ABMIL attention, feature aggregation, global head and global logits, and a weak slice-level lesion localization branch with a slice head, slice logits and per-slice class probabilities over normal, benign and malignant

Research Square preprint (in review)

BE-WISER: Class-Aware Weak Slice-Level Supervision for Breast MRI

Preprint · in review

BE-WISE could say which slice contained something; BE-WISER says what that something is. Each slice inside the Gaussian region around the radiologist's marked slice inherits the exam's class — benign or malignant — while the rest are labelled normal, so the slice head becomes three-class instead of binary. On the latest multicenter ODELIA release (926 / 230 / 292 examinations from six institutions) this reached breast-level AUC 0.864 and an ODELIA score of 0.699, against 0.807 and 0.591 for a breast-level baseline. Localization also improved: Hit@1 rose from 0.554 to 0.632 and the mean slice distance fell from 2.838 to 2.505 slices.

Explainable AIDetection / LocalizationMedical Imaging
1 min read → Paper ↗
Three-stage architecture: an input mammogram through a SigLIP vision encoder and linear projection into visual tokens, a text instruction through a tokenizer into text token embeddings, both combined into a Gemma 3 4B model with 34 transformer layers generating a report autoregressively, then attention weights extracted per token, aggregated across heads and layers and bicubically interpolated to produce one heatmap per generated token

Manuscript in preparation — no venue yet

Explainable Vision-Language Models for Mammography Report Generation

Manuscript in preparation · not peer reviewed

This is the follow-up to MammoBLIP, and it asks a different question: not how good the report is, but whether you can see where each word came from. Domain-adapted SigLIP vision encoders are paired with 4B-parameter LLMs (Gemma 3 and MedGemma), and attention is aggregated across heads and layers to produce one heatmap per generated token. The best of six configurations reached ROUGE-1 0.877 and ROUGE-2 0.843, up 0.127 and 0.143 on MammoBLIP. The more interesting finding is negative: attention often lands on the right microcalcification cluster while the diagnosis is still wrong.

Vision-Language ModelsLLM / GenAIExplainable AI
1 min read →