Biomedical Imaging &
Computer Vision

Core Concepts

What "biomedical imaging" covers (microscopy → histology → CT → MRI)

"Biomedical imaging" in this module spans five physical modalities that share almost nothing except that the input is pixels and the output is biology. Fluorescence microscopy of cultured cells (Cellpose, Cell Painting), confocal hyperspectral live imaging (ESPRESSO), H&E-stained tissue sections at 20–40× magnification (ST-Net, GHIST, HEX, PaSCient, SPARK), multiplexed in situ imaging of RNA and protein at subcellular resolution (CosMx, Xenium, DBiTplus, CODEX), and volumetric clinical scans (MedMPT chest CT). Each modality has its own physics, its own typical image size (224² patches up to 100,000²-pixel whole-slide images), and its own labelling regime.

What unifies them is the AI task pipeline: segment objects (cells, nuclei, organelles, lesions) → classify them (cell type, disease state, grade) → predict something molecular or clinical that the image alone doesn't directly measure (gene expression, protein level, survival, drug response). Cross-link to spatial-transcriptomics.html and spatial-transcriptomics-technologies.html for the molecular substrate that many of these models predict or consume.

The five image-AI sub-tasks (segment / classify / predict modality / generate / multi-agent diagnose)

• Segment. Per-pixel labelling. Cellpose (gradient flows, AP_0.5 = 0.91), StarDist (star-convex polygons, mPQ = 0.5013 on CoNIC), CellSAM (SAM + CellFinder, F1-error 0.40 zero-shot on LIVECell), Baysor (Bayesian + MRF over transcripts, 2× more cells than DAPI-based), ESPRESSO (organelle-level with hyperspectral phasor unmixing). The substrate every other task builds on.
• Classify. Single-cell or whole-slide labelling. StarDist with classification head (6 cell types, 1st place CoNIC 2022), Classpose (8% PQ improvement over Semantic CP-SAM across 6 datasets), PaSCient (135 disease states from 24.3M cells × 5,000 patients, weighted F1 = 0.70).
• Predict modality. Map H&E pixels → molecular readout. ST-Net (DenseNet-121 → 250 genes, RMSE 0.31), Hist2ST (Convmixer + Transformer + GNN, +9% PCC vs HisToGene), GHIST (multi-task UNet 3+ → single-cell SGE, PCC 0.7–0.84), OmiCLIP (CLIP-style, 2.2M H&E–ST pairs), SpatialEx (UNI + hypergraph), HEX (MUSK foundation model → 40 proteins at Pearson r 0.73–0.79).
• Generate. Synthesise perturbed cells or modality-translated images. IMPA (StarGAN-style perturbation prediction, FID +17% over StarGANv2), MolPhenix (CLIP-style retrieval of molecule given image, top-1% recall 77.3% vs 5.1% for CLOOME). See perturbation-modeling.html.
• Multi-agent diagnose. LLM agents that orchestrate analytical pipelines over images. SPARK (1,115 verified parameters from H&E across 5 cancer types and 5,400+ patients, MSI AUROC 0.933). See ai-agents.html.

Why biological images are harder than ImageNet (resolution, batch effects, low labels)

• Resolution mismatch. ImageNet is 224×224 of a single object. A whole-slide H&E image is 100,000×100,000 pixels with 50,000+ cells; a Xenium FOV may contain ~700K transcripts. Patch-based pipelines are mandatory, which means downstream models inherit the patch's locality bias (ST-Net's 100µm spots average over tens of cells; HEX uses 224×224 tiles for 40 protein channels).
• Batch effects dominate. Cell painting plates, microscope channels, scanner colour profiles, and staining protocols all leak signal that swamps biology if not controlled. Spatial transcriptomics across slides has the same problem (Novae explicitly bakes optimal-transport batch correction into pretraining). CellPaint-POSH's CP-DINO needed defocus + coarse-dropout augmentation to handle this; cross-link to sc_batch_correction_integration.html.
• Labels are weak. Pixel-level masks are expensive (Cellpose's 70K+ annotations were a community effort over years). WSI labels are patient-level, not patch-level — that's why PaSCient uses MIL. Spatial-transcriptomics ground truth has only ~3,400 counts per spot in the original ST data; ST-Net trains on 23 patients total. The signal-to-noise is closer to medical imaging than to web data.
• Segmentation errors compound. cellAdmix (Nat Genet 2026) showed every state-of-the-art segmenter (Baysor, BIDCell, ProSeg, 10x Multimodal) leaves cells contaminated with foreign transcripts, and that this contamination drives ~25% of published DIALOGUE/scITD factors and inflated ligand-receptor calls. A 5% segmentation error becomes a fundamentally wrong biological conclusion downstream.

Evolution Timeline

Side-by-Side: Supervised Regression vs Contrastive Pretraining for H&E → Gene Expression

ST-Net and OmiCLIP both map the same H&E tile to a molecular readout, but they diverge on every key design axis: label representation (continuous per-gene counts vs ranked gene-sentence tokens), training objective (MSE regression vs contrastive image–text alignment), and scale (23 patients vs 1,007 samples across 32 organs).

Method Families

Grouped by the AI task and substrate. Within each family, papers are ordered chronologically. Several papers span families (e.g., CellSAM is both segmentation and foundation model); they are placed in the better-fitting family and cross-referenced.

1. Cell & Organelle Segmentation (7)

Per-pixel labelling. The substrate every downstream task depends on. From U-Net + gradient flows (Cellpose) to star-convex polygons (StarDist) to Bayesian transcript-aware models (Baysor) to foundation-model promptable segmentation (CellSAM, Classpose) to organelle-resolution hyperspectral imaging (ESPRESSO) to post-segmentation cleanup (cellAdmix).

2. H&E → Spatial Expression Prediction (6)

Learn the H&E↔molecular bridge. From the seminal CNN ST-Net (250 genes from 100µm spots) through transformer + GNN hybrids (Hist2ST) to single-cell-resolution multi-task UNet 3+ (GHIST), CLIP-style 2.2M-pair foundation models (OmiCLIP), hypergraph H&E-anchored multi-omics (SpatialEx), and virtual proteomics for clinical biomarkers (HEX).

3. Multiplexed Tissue Imaging Technologies & Tools (7)

The platforms that produce multiplexed tissue data and the software that processes them. DSP (44 proteins or 1,412 genes from FFPE ROIs), CosMx SMI (980 RNAs + 108 proteins at subcellular resolution), DBiTplus (sequencing + multiplexed imaging on same section), and the integration ecosystem: Squidpy (technology-agnostic analysis), BANKSY (spatial clustering at scale), CellNEST (GAT-based multi-hop relay LR networks), and the Xenium-Visium-scFFPE breast cancer reference study.

4. Pathology & Spatial Foundation Models (5)

Pretrained encoders trained on massive paired image-omics or image-only datasets that transfer zero-shot to new tissues, panels, and cancer types. CellSAM (Family 1), OmiCLIP (Family 2) are also foundation models; this section features the general-purpose histopathology encoders (UNI, CONCH, GigaPath, PLIP) that most 2024–2026 downstream work — including SpatialEx and HEX elsewhere in this module — builds on, plus Novae, the graph-based spatial-transcriptomics FM. Cross-link to sc-foundation-models.html and pretraining.html.

5. Cell Painting & High-Content Screening (3)

Image-based phenotypic profiling at compound or genetic-perturbation scale. CellPaint-POSH (DINO-ViT on 5-channel + ISS-based CRISPR readout, 1,640-gene druggable genome), MolPhenix (CLIP-style molecule↔image retrieval), IMPA (generative perturbation imagery via StarGAN). Cross-link to perturbation-modeling.html and contrastive-learning.html.

6. Histopathology & Whole-Slide Imaging (1)

WSI-scale analysis. Patient-level disease state from millions of cells across 5,000+ patients via Multi-Instance Learning. HEX (Family 2) and Classpose (Family 1) are also WSI methods; SPARK (Family 7) operates at WSI scale via agents. Cross-link to transfer-learning.html and embedding-strategies.html.

7. Multimodal & Agentic Diagnosis (2)

Image + text + workflow. From paired chest-CT-report vision-language pretraining at clinical scale (MedMPT) to multi-agent LLM systems that autonomously generate, code, and verify analytical hypotheses across cancer cohorts (SPARK). The clinical-deployment edge of biomedical imaging. Cross-link to ai-agents.html, attention-mechanisms.html, retrieval-augmented-generation-bio.html, and reasoning-llms-biology.html.

Selection at a Glance

Quick reference: which imaging recipe to reach for given your task, modality, and label budget.

Reading Order for Newcomers

Practical Implementation Guide

Hands-On Practice

This walkthrough takes you from a raw biomedical image (fluorescence microscopy or H&E) to a segmented, classified result using the core libraries in the field. All snippets are self-contained and can be adapted to DICOM clinical images, NIfTI MRI/CT volumes, or whole-slide SVS files by swapping the loader shown in the Data Structures section.

Environment & packages

Install into a fresh Python 3.10 environment. MONAI covers clinical volumetric imaging; cellpose handles fluorescence/H&E cell segmentation; tiatoolbox is the recommended WSI pipeline; SimpleITK and nibabel cover DICOM and NIfTI I/O respectively.

# conda create -n bioimaging python=3.10 && conda activate bioimaging
pip install torch torchvision torchaudio --index-url https://download.pytorch.org/whl/cu118
pip install monai cellpose stardist scikit-image nibabel SimpleITK pydicom
pip install tiatoolbox openslide-python          # WSI (Homebrew: brew install openslide)
pip install numpy matplotlib tqdm

Hardware. Cellpose and StarDist run comfortably on CPU for small images (<1024×1024). For MONAI 3-D models or WSI tile inference, a GPU with ≥8 GB VRAM is strongly recommended; use torch.cuda.is_available() to verify before starting.

Data structures & formats

• NumPy arrays (H, W, C) or (C, H, W) — the universal in-memory format; channel order matters (scikit-image uses HWC, PyTorch uses CHW).
• DICOM (.dcm) — clinical CT/MRI scans stored slice-by-slice; load with pydicom.dcmread() or SimpleITK.ReadImage(); always read PixelSpacing and SliceThickness metadata — physical voxel size drives all morphology measurements.
• NIfTI (.nii / .nii.gz) — standard for MRI/CT volumes after DICOM conversion; load with nibabel.load(); affine matrix encodes voxel-to-world-space mapping — never ignore it.
• OME-TIFF / SVS (whole-slide images) — multi-resolution pyramidal tiles up to ~100,000×100,000 px; access with tiatoolbox.wsicore.WSIReader or openslide.OpenSlide; always specify the magnification level (e.g., 20×) rather than a pixel level index, as level 0 differs between scanners.
• PNG/TIFF fluorescence stacks — 2-D or 3-D multichannel microscopy; load with skimage.io.imread() (TIFF) or tifffile.imread() for multi-page/hyperstack files.
• Segmentation masks — integer-labelled arrays where 0 = background and each positive integer is a unique cell/instance ID; distinct from semantic masks (class IDs only, no instance distinction).

Minimal code walkthrough

Load a fluorescence image, normalise it, run Cellpose instance segmentation, then measure per-cell mean intensity — a typical starting point for any cell-biology screen.

import numpy as np
from skimage import io, exposure
from skimage.measure import regionprops_table
from cellpose import models
import pandas as pd

# 1. Load — shape (H, W) grayscale or (H, W, C) multichannel
img = io.imread("cells_DAPI.tif")          # uint16 or uint8 fluorescence TIFF

# 2. Normalise to [0, 1] using 1st–99th percentile clip (avoid outlier saturation)
p1, p99 = np.percentile(img, (1, 99))
img_norm = exposure.rescale_intensity(img, in_range=(p1, p99))

# 3. Cellpose segmentation — "nuclei" model for DAPI; set diameter to measured nucleus size in px
model = models.Cellpose(gpu=True, model_type="nuclei")
masks, flows, styles, diams = model.eval(
    img_norm,
    diameter=30,        # estimate from scale bar or CellProfiler "MeasureObjectSizeShape"
    channels=[0, 0],    # grayscale: [chan_of_interest, chan_for_nucleus_or_0]
    flow_threshold=0.4,
    cellprob_threshold=0.0,
)

# 4. Measure per-cell features using regionprops
props = regionprops_table(
    masks,
    intensity_image=img_norm,
    properties=["label", "area", "mean_intensity", "eccentricity", "bbox"],
)
df = pd.DataFrame(props)
print(df.head())

# 5. Quick QC — remove cells touching the image border and very small debris
border_labels = set(np.unique(masks[0, :])) | set(np.unique(masks[-1, :])) \
              | set(np.unique(masks[:, 0])) | set(np.unique(masks[:, -1]))
df = df[~df["label"].isin(border_labels) & (df["area"] > 100)]
print(f"Cells passing QC: {len(df)}")

Common pitfalls & tips

• Always read pixel spacing / voxel size metadata. DICOM PixelSpacing and NIfTI affine matrices encode physical scale. Skipping them means your area, diameter, and volume measurements are in pixel units — meaningless across scanners or magnifications. Use SimpleITK.GetSpacing() or nibabel.load().header.get_zooms() to extract it before any morphology measurement.
• Set Cellpose diameter correctly or let it estimate. Leaving diameter=None triggers auto-estimation but adds ~20 s per image; too-small values over-segment, too-large values merge nuclei. Measure 10–20 representative cells with ImageJ/FIJI first, then fix the diameter for reproducibility across a batch.
• Tile WSIs at the right magnification and with overlap. In tiatoolbox, always request tiles at a named magnification (e.g., power=20) rather than a pyramid level. Use at least 64 px overlap on each edge and stitch predictions with weighted blending or Gaussian feathering — hard-edge tiling produces grid artefacts in both segmentation masks and feature maps.
• Normalise H&E stain before deep models. Macenko or Vahadane stain normalisation (available in tiatoolbox.tools.stainnorm) reduces scanner-to-scanner colour variation by ~40% on CoNIC-style benchmarks; without it, a model trained on one scanner fails on another. Apply normalisation before any patch extraction step.
• Patient-level data leakage is the most common evaluation error. If a patient has 50 WSI tiles and you randomly split tiles into train/test, the model memorises patient appearance. Always split at the patient (or slide) level; use stratified splits to balance disease class across folds.
• Class imbalance in histology is severe. In CoNIC, neutrophils are <2% of nuclei; in PaSCient, rare disease classes have <50 patients. Use class-weighted sampling, Focal loss, or Focal Tversky loss (as in Classpose) rather than standard cross-entropy, and always report per-class F1 rather than macro accuracy — a model predicting "lymphocyte" for every nucleus can look 70% accurate while being useless.