Generative Foundation Models for Gigapixel Imaging
Event Description
Abstract: Generative models have transformed natural image synthesis, driven by billion-scale datasets of image-caption pairs. Specialized domains, such as digital histopathology and satellite imagery, differ in two ways: their images reach gigapixel resolution, far beyond any feasible training resolution, and they lack paired captions, since captioning the patches of a single whole-slide image (WSI) takes about 40 hours of pathologist time. Overcoming these challenges would enable generative models that create teaching cases of rare diseases, simulate tumor progression, and serve as a foundation for digital twins.
In this thesis, we build generative foundation models for these domains, scaling from small patches to gigapixel images without patch-level annotations. We begin with PathLDM, which conditions histopathology patch generation on
slide-level pathology reports, enabling the first text-to-image generation in histopathology. Since reports do not describe individual patches, we propose LRDM, which uses embeddings from pretrained self-supervised learning (SSL) encoders as patch-level conditioning for both histopathology and satellite images. ZoomLDM extends SSL conditioning across eight magnifications with a single model, capturing structure from individual cells to tissue architecture. To scale across cancer types, we present PixCell, the first generative foundation model for histopathology, trained on 30.8 million patches from 69,184 WSIs across 28 organ types. Synthetic images from PixCell can stand in for real data in SSL training, enable privacy-preserving data sharing, and improve downstream tile-level classifiers. Extending this to gigapixel scale, we introduce BigCell, which generates coherent WSIs of up to 32768x32768 pixels by guiding a 2048x2048 flow-matching transformer with a grid of foundation model embeddings, and speeds up generation by over 3x with an importance-based schedule. BigCell enables text-to-image synthesis at 8192x8192 resolution, with synthetic samples improving slide-level few-shot classification by up to 18%. Beyond pathology, our SSL-conditioned models serve as data generators for self-supervised learning on natural images. Finally, to make gigapixel synthesis practical, we plan to reduce BigCell's generation time from about 5 hours to under 20 minutes per 32768x32768 image through few-step distillation and efficient attention, and to extend text-to-image synthesis from 8192x8192 regions to entire slides with a pretrained language model.
Speaker: Srikar Yellapragada
Location: NCS 220
In this thesis, we build generative foundation models for these domains, scaling from small patches to gigapixel images without patch-level annotations. We begin with PathLDM, which conditions histopathology patch generation on
slide-level pathology reports, enabling the first text-to-image generation in histopathology. Since reports do not describe individual patches, we propose LRDM, which uses embeddings from pretrained self-supervised learning (SSL) encoders as patch-level conditioning for both histopathology and satellite images. ZoomLDM extends SSL conditioning across eight magnifications with a single model, capturing structure from individual cells to tissue architecture. To scale across cancer types, we present PixCell, the first generative foundation model for histopathology, trained on 30.8 million patches from 69,184 WSIs across 28 organ types. Synthetic images from PixCell can stand in for real data in SSL training, enable privacy-preserving data sharing, and improve downstream tile-level classifiers. Extending this to gigapixel scale, we introduce BigCell, which generates coherent WSIs of up to 32768x32768 pixels by guiding a 2048x2048 flow-matching transformer with a grid of foundation model embeddings, and speeds up generation by over 3x with an importance-based schedule. BigCell enables text-to-image synthesis at 8192x8192 resolution, with synthetic samples improving slide-level few-shot classification by up to 18%. Beyond pathology, our SSL-conditioned models serve as data generators for self-supervised learning on natural images. Finally, to make gigapixel synthesis practical, we plan to reduce BigCell's generation time from about 5 hours to under 20 minutes per 32768x32768 image through few-step distillation and efficient attention, and to extend text-to-image synthesis from 8192x8192 regions to entire slides with a pretrained language model.
Speaker: Srikar Yellapragada
Location: NCS 220
