AI in Medicine: Learning from Incomplete Patient Data (IN2107)

Course details

Clinical decisions rarely rely on a single source of information. A patient may be described by medical images, radiology reports, laboratory values, genomics, and clinical history. In practice, however, these modalities are heterogeneous, noisy, and frequently incomplete. This seminar explores how machine learning can build useful multimodal patient representations under these conditions.

We study principal fusion strategies, including early, intermediate, and late fusion, joint embeddings, and cross-attention architectures. Particular attention is given to vision-language models and to methods that remain robust when modalities are missing during training or inference. Each project is anchored to a concrete clinical endpoint, such as diagnosis, prognosis, or treatment-response prediction.

Key topics to be covered include:

• Multimodal patient representation learning from heterogeneous, noisy, and incomplete clinical data
• Early, intermediate, and late fusion
• Joint embeddings and cross-attention architectures
• Vision-language models for medicine
• Learning with missing modalities at training and inference time
• Evaluation for clinical endpoints such as diagnosis, prognosis, and treatment-response prediction

Requirements:

• Prior experience and a good understanding of machine learning and statistics
• Very good programming skills in Python and PyTorch
• Interest in medical imaging and multimodal data

Learning and project format:

• The seminar centers on a hands-on implementation project using a curated dataset and a provided PyTorch framework.
• Students work in teams of two. Each team selects a research paper, with suggestions provided by the lecturers, and implements, trains, evaluates, and analyzes a method on the provided data.
• An introductory session covers multimodal learning, fusion strategies, critical reading of research papers, and preparation of a scientific poster.
• Invited talks by clinical collaborators and academic experts provide perspectives on clinical motivation, medical imaging, and current research developments.
• Each team gives an intermediate oral presentation.
• The seminar concludes with a conference-style poster session in which teams present their implementations and results.

Lecturers:

• Cosmin Bercea
• Keno Bressem
• Ha Young Kim
• Julia Schnabel