Post-Training for Biological Foundation Models

🎯 Core Concepts

Why post-training is now where the action is

Pretraining gives you a substrate; post-training gives you a tool. The 2023–2024 wave of biological foundation models — Geneformer, scGPT, scFoundation, ESM-2, Nucleotide Transformer — established that masked-LM, contrastive, and decoder-only objectives can absorb tens to hundreds of millions of cells or sequences. But every downstream application that matters — annotate a disease atlas, predict a perturbation, design a protein, draft a cardiology consult — requires a second training phase: aligning, specialising, verifying, or compressing the base model so it behaves correctly on the actual task.

This module is the umbrella view: given a base biological FM, which post-training recipe should you reach for, and why? Focused companions on this hub cover the pieces in depth — fine-tuning (LoRA/PEFT/adapters), pretraining (what comes before), generative models (the architectures being post-trained), and transformer architecture (the blocks underneath).

The post-training stack: 7 composable stages

• 1 · Supervised fine-tuning (SFT) / domain adaptation. Update some (or all) base weights on labeled task data. The dominant 2023–2024 pattern: scGPT-Protocol (99.5% F1 retinal annotation), CodonFM continued pretraining, TIME-LLM reprogramming, DANCE dual-modality adaptation.
• 2 · Parameter-efficient fine-tuning (PEFT). Freeze the base, train <1–5% of parameters via LoRA, prefix tokens, or autoencoders. scPEFT: 96–99.95% parameter reduction with gains over full FT; PEFT-Proteomics: within 10–15% of full FT at 240–395× fewer params.
• 3 · RL elicitation of reasoning. R1-style: reward verifiable reasoning chains. Cell-o1 (o1 distillation + GRPO, +73% on CellPuzzles), Gene-R1 (3-stage pipeline, 108–257% ROUGE-L over base), ProteinRL (prior-regularised policy gradient), C2S-Scale (GRPO + BioBERTScore).
• 4 · Multi-agent verification. Wrap the base model in structured workflows with database lookups, self-critique, and confidence scoring — "post-training without gradient updates". GeneAgent (44% → 19% hallucination), CASSIA, CyteType, SpatialAgent, SPARK, AMIE.
• 5 · Memory and active learning. Long-running deployments need a memory layer (Mem0: −91% p95 latency, >90% token-cost reduction) and/or an experimental feedback loop (DeMeo: closed-loop ARL, 17×/13× hit-rate over random).
• 6 · Quantization & compression. Trade precision for speed and memory. Geneformer-Scaling 4-bit NF4 + double quantization preserves contextual embeddings at r=0.998; in-silico KO screen cost drops $25k → <$5k.
• 7 · Inference-time compute scaling. Chain-of-thought sampling, best-of-N, MCTS-guided rollout — adapt the model's reasoning budget, not its weights. AMIE's conversational refinement loop is Stage 7 post-training in deployment: zero gradient updates, all the adaptation happens at inference time via multistep reasoning, retrieval, and self-critique. DeepSeek-R1 proved the pattern generalises far beyond biology. Prerequisite: a verifiable scoring function that can tell a better reasoning chain from a worse one — without it, more inference-time compute just samples more confidently wrong answers.

Bio-specific constraints

• Verifiable rewards are rare. Math has ground truth; biology mostly doesn't. Cell type labels disagree across atlases by ~25%, and mechanism hypotheses have none. The successful bio-RL papers all engineer a verifiable proxy — batch-level one-to-one constraints (Cell-o1), semantic + lexical rewards against curated labels (Gene-R1), or computable properties like charge and pLDDT (ProteinRL). If you can't define one, RL elicitation will reward-hack.
• Data is small and uneven. Most biological tasks live in 1K–100K labelled examples — exactly the regime where PEFT beats full fine-tuning, because the full model overwrites itself.
• Evaluation is expensive and benchmarks lag. Wet-lab validation takes weeks to months. Multi-agent verification works around it with database lookups and calibrated LLM judging as a cheap proxy — improving the verification surface often matters more than improving base accuracy.
• Catastrophic forgetting is real. Fine-tune a 33M-cell scFM on your 5K-cell disease atlas and you can erase the pretrained gene–gene relationships that made it useful. Every successful paper has an explicit anti-forgetting mechanism.
• The base model is increasingly fungible. CyteType's finding is that performance is architecture-driven, not LLM-driven — open-weight backbones reach 95% of frontier at a fraction of cost inside a good scaffold. The post-training recipe is now the moat.

🕰 Evolution Timeline

1. 2017 · PPO and the RL substrate. Proximal Policy Optimization defines the policy-gradient template every later RL fine-tune uses; REINVENT ports it to chemistry with prior-regularised loss — the direct ancestor of ProteinRL.
2. 2023 · ProteinRL, the first bio RL fine-tune. REINVENT-style policy gradient on ProGen2-764M for property-directed protein design — β-lactamase net charge pushed to +18/−39 while preserving fold and active-site residues. Same year, InstructGPT-style RLHF goes mainstream in NLP.
3. 2024 · PEFT mainstreams; SFT protocols mature. LoRA comes to ESM-2 with proteomics-specific hyperparameters (r≥4, W_K+W_V); DANCE tackles dual-label-scarcity multi-omics; CodonFM scales codon-level masked LM; TIME-LLM reprograms a frozen Llama-7B with an input head only.
4. 2025 H1 · Multi-agent verification + RL reasoning land in biology. scGPT-Protocol standardises full-FT (99.5% F1); Mem0 defines the production memory layer; SpatialAgent launches the autonomous-agent template; Cell-o1 ports o1 distillation + GRPO to single-cell (+73%); GeneAgent cuts hallucination 44% → 19%.
5. 2025 H2 · PEFT for scLLMs, community ecosystems, active learning. Gene-R1's 3-stage RL pipeline; ColabSaprot ships SaprotHub community adapters; DeMeo demonstrates closed-loop ARL (17× hit-rate); CyteType proves architecture over model; CASSIA ships 5-agent calibrated annotation; scPEFT systematises four adapter types.
6. 2026 · Clinical RCTs, quantization-native scFMs, agentic discovery. AMIE delivers the first RCT-level evidence that inference-time post-training improves cardiologists (46.7% vs 32.7% preferred); Geneformer-Scaling defines scaling laws to 316M and ships 4-bit QLoRA; SPARK runs autonomous concept discovery across 5,400+ patients for ~€4K.

⚖️ Side-by-Side: PPO vs DPO vs GRPO

Three ways to turn human or verifiable feedback into a better policy — PPO (online RL with an explicit reward model), DPO (offline, direct preference optimization, no reward model), and GRPO (group-relative policy optimization on verifiable rewards, no reward model, no value function). The same starting point — a base policy π_θ and preference data or a verifiable reward — becomes three very different training pipelines.

The split traces back to one root decision — whether to learn a separate reward signal or eliminate it. PPO pays for training and maintaining an RM but handles any preference signal; DPO eliminates the RM by reparameterising the reward as a log-ratio between current and reference policy — purely offline, but vulnerable to distribution shift; GRPO removes both RM and value function by computing advantage within a sampled group, which only works when the reward is checkable without a neural model — exactly the verifiable-reward design criterion that makes Cell-o1, Gene-R1, and C2S-Scale tick in biology.

Three variants worth knowing before picking one of these three. ORPO (Hong et al. 2024) drops the reference model entirely and optimises an odds-ratio term added directly to the SFT loss — one training stage instead of SFT-then-DPO, and more stable than DPO in small-data regimes exactly like the <100K-label tasks common in biology. SimPO (Meng, Xia & Chen, NeurIPS 2024) is also reference-free, but its fix targets DPO's length bias: it normalises the reward by sequence length so the policy can't win by generating longer (not better) completions — worth checking whenever you eval by win-rate. DAPO (ByteDance Seed, 2025) instead targets GRPO's training instability at scale — Clip-Higher, dynamic sampling, token-level policy-gradient loss, and overlong-reward shaping — and has become the de facto reference recipe for open, reproducible GRPO training since C2S-Scale and Cell-o1-style pipelines were built.

🧩 Method Families

The 22 papers fall into six families by post-training recipe. Some papers fit two — placed in best fit.

1 · Supervised Fine-Tuning & Domain Adaptation

Classical post-training: cross-entropy/NLL on labelled task data, updating some or all base-model weights. The workhorse pattern when you have curated labels and a base model in the right modality.

2 · Parameter-Efficient Post-Training (PEFT)

Freeze most of the base model; train low-rank adapters, prefix tokens, or autoencoder bottlenecks. The default recipe when you have <100K labelled examples or a single consumer GPU. Code-level deep-dive in the fine-tuning module.

3 · RL Elicitation of Reasoning

Following the R1 / GRPO template, reward verifiable correctness on structured reasoning traces. Works when you can define a checkable reward — a computable property, an exact-match label, a multi-objective score.

4 · Multi-Agent Verification as Post-Training

The base model is left untouched; the system wraps it with structured workflows, database lookups, self-critique, and confidence scoring. This is "post-training without gradient updates" — the recipe when you can't (or shouldn't) fine-tune the base model.

5 · Memory, Active Learning & Online Refinement

Post-training that runs during deployment — memory systems that consolidate facts across sessions, active learning loops that query the model for the next experiment, and review syntheses tying the agentic discovery pattern together. The bridge from offline fine-tune to a continually improving system.

6 · Clinical / Translational Post-Training

The hardest test: does the post-training recipe survive an RCT with real patients? AMIE is the first paper to clear this bar in a cardiology subspecialty.

🧭 Selection at a Glance

Which post-training recipe to reach for, given your base model, data scale, compute budget, and downstream goal. Scope: the table covers the 11 core recipes; the family cards above carry the full paper details.

📚 Reading Order for Newcomers

1. PEFT-Proteomics (2024, PNAS). The cleanest, most rigorous PEFT paper in biology. Read for LoRA mathematics, domain-specific hyperparameter tuning (r≥4, W_K+W_V), and the unsettling finding that frozen-MLP often beats both PEFT and full FT. Sets the right expectations.
2. scPEFT (2025, Nat Mach Intell). The systematic extension to single-cell LLMs. Four adapter types, 96–99.95% parameter reduction, anti-forgetting argument. Read alongside PEFT-Proteomics to see how the recipe transfers across modalities.
3. Cell-o1 (2025, arXiv). The clearest worked example of R1-style post-training in biology. Two-stage SFT distillation + GRPO with a verifiable batch-level reward. Read this before the more elaborate Gene-R1 pipeline; it makes the GRPO mechanics concrete.
4. GeneAgent (2025, Nat Methods). The simplest, most effective multi-agent verification recipe. 4-stage Generate → Verify → Modify → Summarize loop with database queries. Hallucination 44% → 19% with zero gradient updates. The intro to "post-training without training."
5. Geneformer-Scaling (2026, Nat Comput Sci). The synthesis paper. Defines transcriptomic scaling laws, then layers 4-bit QLoRA on the 316M model. Read for the quantization-preserves-contextual-embeddings argument (r=0.998 on GATA4 KO) — this is what makes scFM post-training economically viable.
6. AMIE (2026, Nat Med). The clinical reality check. No fine-tuning, no PEFT — pure inference-time post-training via multistep inference + self-critique + web search + conversational refinement. First RCT-level evidence in a cardiology subspecialty. Read to recalibrate what "good enough" means in deployment.
7. DeMeo / DrugReflector (2025, Science). The only paper in this module with a real wet-lab feedback loop: 17× hit-rate vs random for megakaryocyte differentiation, with the biological signature refined round-over-round by experimental data. Read last, to understand the gap between offline post-training — every other recipe above, however sophisticated — and a system that keeps improving from real experimental results after deployment.

🧰 Practical Implementation Guide

🛠 Hands-On Practice

This walkthrough takes you from a raw base model to a preference-aligned checkpoint using the trl library. The DPO path is the most accessible entry point into post-training: it skips an explicit reward model and optimises directly on chosen/rejected pairs — a practical first step before exploring online methods like PPO or GRPO.

Environment & packages

All packages below are available on PyPI. vllm is optional but strongly recommended for GRPO and PPO, where fast rollout generation is the main bottleneck.

# create a clean env first (Python 3.10+)
pip install torch torchvision --index-url https://download.pytorch.org/whl/cu121
pip install transformers==4.45.0 datasets peft accelerate
pip install trl==0.11.4           # SFTTrainer, DPOTrainer, GRPOTrainer, PPOTrainer
pip install deepspeed             # multi-GPU ZeRO-2/3 for larger runs
pip install vllm                  # fast rollout engine for online RL (PPO/GRPO)

Hardware. DPO on a 7B model with LoRA (rank 64) fits comfortably on a single 40 GB A100 or two 24 GB consumer GPUs with accelerate; full-parameter DPO needs 2–4× A100 80 GB. GRPO with vllm rollouts requires a separate GPU for the rollout engine (8+ GB) alongside the training GPU.

Data structures & formats

• Preference dataset — three columns: prompt (string), chosen (string or messages list), rejected (string or messages list). Standard hub example: trl-lib/ultrafeedback_binarized.
• SFT dataset — a messages column containing a list of {"role": ..., "content": ...} dicts; the tokenizer's chat template converts this to input IDs + labels. Alternatively a flat text column with the template already applied.
• LoRA adapter — a tiny adapter_config.json + adapter_model.safetensors written by peft; merged into base weights at inference with model.merge_and_unload().
• Reward / verifiable signal (GRPO) — a Python function reward_fn(completions, **kwargs) -> list[float] called on each group of rollouts; return scalars — no gradient required.
• GRPO group rollouts — for each prompt, num_generations completions are sampled (default 8); the per-group mean reward is subtracted as a baseline before computing the policy-gradient loss.

Minimal code walkthrough

A complete DPO run: load a base model + LoRA, attach a preference dataset, configure DPOConfig, train for one epoch, and save the adapter.

from datasets import load_dataset
from peft import LoraConfig, TaskType
from transformers import AutoModelForCausalLM, AutoTokenizer
from trl import DPOConfig, DPOTrainer

# 1. Load base model + tokenizer
model_id = "Qwen/Qwen2.5-7B-Instruct"
tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(
    model_id, torch_dtype="auto", device_map="auto"
)

# 2. LoRA config — keeps base frozen; ~0.5% trainable params
peft_cfg = LoraConfig(
    task_type=TaskType.CAUSAL_LM,
    r=64, lora_alpha=128, lora_dropout=0.05,
    target_modules=["q_proj", "v_proj", "k_proj", "o_proj"],
)

# 3. Preference dataset (prompt / chosen / rejected columns)
dataset = load_dataset("trl-lib/ultrafeedback_binarized", split="train")

# 4. DPO hyperparameters
# beta controls KL penalty: higher beta = stay closer to reference policy
training_args = DPOConfig(
    output_dir="./dpo-out",
    num_train_epochs=1,
    per_device_train_batch_size=2,
    gradient_accumulation_steps=8,
    learning_rate=5e-5,
    beta=0.1,                   # KL regularisation coefficient
    max_length=1024,
    max_prompt_length=512,
    bf16=True,
    logging_steps=10,
    save_strategy="epoch",
)

# 5. DPOTrainer handles reference model internally (frozen copy of model)
trainer = DPOTrainer(
    model=model,
    args=training_args,
    train_dataset=dataset,
    processing_class=tokenizer,
    peft_config=peft_cfg,
)
trainer.train()
trainer.save_model("./dpo-adapter")

Common pitfalls & tips

• Always SFT before preference optimisation. DPO and GRPO assume the policy already generates coherent completions. Starting from a raw pretrained base (not instruction-tuned) causes extremely noisy gradients because the chosen/rejected distinction is meaningless relative to random outputs. Run at least a light SFT pass with SFTTrainer first.
• Chat-template / tokenization mismatch is the silent killer. The prompt in your preference dataset must be tokenized with the same chat template the model was trained with — including special tokens, role labels, and any BOS/EOS placement. A mismatched template shifts where labels start and produces near-zero DPO loss that looks like convergence but is not.
• Tune beta carefully in DPO. beta is the coefficient on the KL divergence term that keeps the trained policy close to the reference. Too low → reward hacking; too high → the model never moves from the reference. Start at 0.1 and sweep 0.05–0.5.
• Length bias inflates win-rate. DPO and GRPO reward models (including LLM judges) systematically prefer longer completions. Track response length alongside reward; add a length-penalty term to the reward function, or filter your preference dataset to be length-balanced before training.
• KL explosion in online RL (PPO/GRPO) kills training. Monitor kl_coef and the actual KL divergence at every logging step. If KL spikes sharply after a few hundred steps, reduce the rollout learning rate or increase kl_coef. vllm rollout speeds up generation 5–10× but requires keeping the rollout engine in sync with updated policy weights.
• Eval beyond loss: use win-rate and downstream benchmarks. DPO loss decreasing monotonically does not mean the model is improving. Run periodic win-rate evaluations (compare sampled completions against the reference model with an LLM judge or a rule-based verifier), and track a fixed held-out benchmark to catch regressions on general capability.

🔗 Related Modules in This Hub

• Fine-Tuning & Adaptation — the code-level deep-dive on LoRA/PEFT/adapters that this module surveys from above (the retrofit story: head tuning, LoRA packs, and load-bearing walls).
• Pretraining & Foundation Models — what comes before: the masked-LM and contrastive objectives that produce the substrate this module post-trains.
• Generative Models — the VAE / diffusion / flow / GNN architectures that RL elicitation and ProteinRL fine-tune for design tasks.
• Transformer Architecture — the power brick underneath everything on this page: attention, residuals, norms, and the encoder/decoder variants being adapted.