Advancing AI for WeaklySupervised Multimodal Alignment and Query-Based Interpretation in Biological Imaging

About This Grant

PROJECT SUMMARY A major challenge in modern biological data analysis is integrating and reasoning over the vast volumes of unstructured, multimodal data now available, such as images and text. While each modality offers complementary biological insight, they are not encoded in a shared format or language, making it difficult to align them or reason across them computationally. A core unmet need is the development of AI systems that can bridge this divide by learning shared representations across data types. This project targets building AI systems for aligning and reasoning jointly over biological image and text data. We focus on microscopy and the challenge of interpreting microscopy images often requiring integration with broader biological context—relating observed phenotypes to those seen in other experiments, identifying plausible mechanisms, and connecting to relevant prior studies. This knowledge is frequently buried in unstructured images and text scattered across publications, databases, and annotations. Conventional AI systems rely on supervised learning, which demands large amounts of manually annotated data and cannot scale to the complexity or breadth of modern biology. Training AI models using weak supervision offers a promising alternative: by learning from loosely aligned image-text pairs, models can capture cross-modal associations from noisy but abundant sources. Vision-language models (VLMs) built on this principle embed images and text into a shared semantic space and support flexible reasoning tasks such as retrieval and question answering. However, current models often suffer from “blurry vision”—they can identify broad semantic matches between images and text but fail to resolve the fine-grained visual distinctions essential for biological interpretation. The goal of our project is to overcome this limitation by advancing weak supervision methods that enable fine-grained alignment between biological images and text, with a focus on microscopy. We will curate a large and diverse dataset of fine-grained image-text pairs and train a visual encoder using multi-scale contrastive learning to integrate both global and local alignment signals. This encoder will power an agentic AI system for query-based interpretation of microscopy images, that can retrieve relevant biological evidence and generate natural-language interpretations of microscopy images in response to researcher queries. We will validate the system in expert-driven use cases spanning single-cell perturbation and tissue-level pathology, and disseminate it through integration into widely used imaging workflows. By building AI tools that help researchers connect microscopy image content to pathways, phenotypes, and prior studies, we aim to support flexible, biologically grounded exploration and accelerate data-driven discovery. By open-sourcing our datasets, methods, and trained models for fine-grained image-text alignment, we also aim to advance the broader capabilities of multimodal AI for biological data analysis.