From General to Clinical - Adapting Foundation Models for Medical Images (IN2107)

Course details

Foundation Models in Deep Learning for Medical Imaging Deep learning models, from CNNs to Vision Transformers, have achieved impressive results for a variety of benchmarks and applications. However, real-world deployment, especially in medical imaging, remains challenging due to unseen variations in scanners, patient demographics, and pathological conditions.

To address this issue, recent work has focused on foundation models: large-scale models pretrained on diverse data using self- or weakly supervised learning. CLIP-based models learn general-purpose representations that transfer well to downstream tasks. For the context of medical imaging, specific fine-tuned models such as MedSAM and BiomedCLIP demonstrate strong potential.

However, adapting foundation models to new domains or tasks often remains challenging. Although domain generalization and adaptation approaches exist, they may overfit or require target data. A recent paradigm, test-time adaptation, mitigates this by adapting alongside inference to unseen data. The techniques include adapting dynamically through model updates or prompt modifications that support more flexible deployment.

This seminar will explore the foundations and capabilities of foundation models, with a focus on pretraining strategies, representational generalization, and efficient adaptation methods. We will examine state-of-the-art foundation models and discuss how they can be adapted to downstream tasks, especially in medical imaging, through techniques such as self-supervised learning, cross-domain transfer, and parameter-efficient tuning.

Key topics to be covered include:

• Introduction to Self Supervised Learning and Foundation Models
• Adaptation techniques to clinical applications
• Examples of Foundation models in medical imaging
• State-of-the-art methods

Requirements:

• Background in image processing and machine learning/deep learning
• Interest in medical image analysis anf foundation models
• Interest in research

Please register via the TUM matching system: https://matching.in.tum.de.

Additional information can be found here.

Intro slides: