From the original Attention Is All You Need to the zoo of encoder, decoder, and hybrid designs powering
single-cell, genomic, and protein foundation models — the blueprint behind the biology.
5
Core Architectures
20+
Biology Models
4
Biological Domains
2017–2026
Year Range
LEGO lens · transformer architecture
电池盒、电线、马达、齿轮组、结构框架——拼出会思考的机器。
Every motorized LEGO build runs on the same five-part power unit: battery box, wires, motor, gears, frame. Transformer architecture is the same kit for sequence models — an input embedding to supply "energy," residual connections to wire the layers together, self-attention as the motor, a feed-forward network as the gearbox, and layer norm as the frame that keeps the whole build from rattling apart.
🧱 The LEGO Power Unit: Transformer Architecture, Part by Part
「不同的零件，各司其职；组合在一起，才有强大的动力。」
Open any LEGO Power Functions set and five parts fall out: a battery box, wires, a motor, a set of gears, and a Technic frame to hold it all together. Each does exactly one job, and none of them move a vehicle alone. Line them up next to a transformer block and the correspondence is one-to-one — same idea, different world.
Part 1 · 电池盒 Battery box
Input embedding
The battery box supplies electricity — without it, nothing in the build has any power to move at all. It's the first thing you snap in, before any other part matters.
Input embedding. token / gene / cell → embedding vector. Converts the raw outside world (text, gene expression) into a vector the model can compute on — the "energy" that feeds the whole network. See input embedding ↓
Part 2 · 电线 Wires
Residual connection
Wires carry power from one unit to the next and link the pieces together — a "shortcut" so current from the battery box reaches the motor no matter how the build is arranged in between.
Residual connections. Carries information from one layer straight to the next, a shortcut that keeps signal from being lost as it passes through deep stacks. See residuals ↓
Part 3 · 马达 Motor
Self-attention
The motor turns electricity into motion — the core driving force of the whole build. Everything downstream depends on what the motor decides to do with the power it's given.
Self-attention. Lets every position interact with every other position, deciding what to "pay attention to" — the core dynamic force of a Transformer. See attention ↓
Part 4 · 齿轮组 Gears
Feed-forward network (MLP)
The gear train regulates speed and torque, controlling exactly how the motor's output actually reaches the wheels.
Feed-forward network. Linear → Activation → Linear. Applies a non-linear transform to the attention output, boosting the model's representational power — accelerating and regulating the flow of information. See the FFN ↓
Part 5 · 结构框架 Technic frame
Layer normalization
The frame holds every part in its place and keeps the overall structure steady — without it, the whole assembly would rattle apart under its own vibration.
Layer normalization. Stabilizes every layer's output so the signal never blows up or vanishes, letting the whole model run steadily. See normalization ↓
Part 6 · 完整乐高动力模型 The complete power model
Transformer block × N
All the units working together drive the vehicle forward. Pull out any single part and the whole build stalls — no wires, no motor, no motion.
Transformer. Self-attention → Add & Norm → Feed-forward → Add & Norm, stacked N times: input tokens go in, meaningful output representations come out. See the full block ↓
The one-to-one mapping. 乐高动力单元 (LEGO Power Unit) → Transformer Architecture → 对应关系与作用 (correspondence & role) — the reference diagram the six parts above are drawn from.The power unit in four plays. Battery box in, energy for everything (input embedding) · wires linking every unit (residual connections) · the motor as core drive (self-attention) · gears regulating the output (feed-forward network) — held together by the frame (layer norm).
Same idea. Different world. LEGO builds cars. Transformers build intelligence.
The Blueprint Story
Why architecture matters as much as the training objective
You can have the perfect tokenization strategy and the right pretraining objective, but if the
architecture routes information incorrectly, the model cannot learn what you need it to learn.
The architecture decides three things: which tokens can see which other tokens
(attention mask), how information is transformed at each layer (the transformer
block), and what positional information the model receives about token order
or biological coordinates.
In biology, this translates to concrete design choices: an encoder-only model sees the whole gene
expression profile at once (good for classification), a decoder generates one gene at a time
(good for simulation), and an encoder-decoder translates one biological state to another
(perturbation prediction, cross-species transfer). Getting this choice wrong is not a minor
performance hit — it changes what the model can and cannot represent.
📖 Encoder-Only
Bidirectional context. Best for understanding / representation. Geneformer, DNABERT-2, ESM2.
✏ Decoder-Only
Causal / autoregressive. Best for generation & simulation. TranscriptFormer, scGPT.
⇄ Encoder-Decoder
Source → Target. Best for seq-to-seq translation. T5, some perturbation models.
🌀 SSM / Hybrid
Sub-quadratic. Best for very long genomic sequences. Mamba, Hyena, Evo 2.
Every transformer in biology — from BERT to GPT to AlphaFold2 — is built by stacking this same module
repeatedly (6×, 12×, 24× ...). The block has two sub-layers: Multi-Head Attention (which lets tokens
talk to each other) and a Feed-Forward Network (which transforms each token independently). Both are
wrapped with Layer Normalization and a residual connection that adds the
input directly to the sub-layer output. The residual is what allows very deep networks to train stably.
LEGO analogy: Think of the transformer block as a universal LEGO adapter brick.
Every LEGO creation uses the same adapter regardless of what it's building — a spaceship or a castle.
The MHA is the part that checks how each brick connects to every other brick in the build; the FFN is
the part that reshapes each individual brick. The residual is the guarantee that you can always
snap the original brick back on if the reshaping goes wrong.
Variant
What changes
When to use
Bio example
Post-LN (original)
LayerNorm after residual add
Original BERT/GPT; needs careful LR warmup
Original Transformer (Vaswani 2017)
Pre-LN
LayerNorm before each sub-layer
More stable training; default in modern models
Geneformer, scGPT, most 2022+ models
Parallel FFN+Attn
FFN and MHA run simultaneously
Faster training at scale; ~15% throughput gain
GPT-J, some large protein models
SwiGLU FFN
Gated linear unit replaces ReLU
Better representation learning per param
LLaMA, ESM3, many 2024+ FMs
Same block, four wiring diagrams. 乐高动力单元接线方式 (LEGO Power Unit Wiring) vs Transformer Block Variant vs 作用与特点 (Role & Intuition) — Post-LN, Pre-LN, Parallel FFN+Attn, and SwiGLU FFN, side by side.
2
Encoder-Only Architecture
Bidirectional self-attention — every token sees every other token
In an encoder-only model, every token can attend to every other token in both directions —
past and future. The output is a dense representation of each input token that encodes its
meaning in context of the whole sequence. This is ideal whenever the task requires
understanding a complete biological entity: classifying a cell type, scoring a gene's regulatory
importance, or embedding a protein for downstream search.
LEGO analogy: A LEGO expert who lays out all the bricks on a table and
examines every brick in the context of every other brick simultaneously — before deciding what anything
means. Gene A's representation knows about Gene E because the expert could see them both at the same time.
Predict whether a sampled protein was expressed in the cell
UCE (Rosen/Roohani et al. 2026, Nature)
Cross-species zero-shot cell embedding, atlas building
🧬 Case study: UCE's genome-agnostic tokenizer
Universal Cell Embedding (UCE) is a 33-layer, 650M-parameter encoder trained
self-supervised on 36M cells spanning 8 species (CellXGene corpus). Its trick isn't in the
attention mechanism — it's in the token itself. Instead of a learned per-gene embedding table
(which only knows genes seen during training), each gene token is its ESM2 protein-language-model
embedding, with genes grouped by chromosome and sorted by genomic location, then sampled
with replacement weighted by expression level. The masked objective is binary, not regression:
"was protein Gp expressed in cell C?"
Because any gene's identity is defined by what its protein looks like rather than an
arbitrary vocabulary index, UCE embeds species it never trained on — green monkey, naked mole
rat, chicken, even Drosophila — with no fine-tuning, and this zero-shot embedding
space is what powers the paper's 36M-cell Integrated Mega-scale Atlas (IMA).
3
Decoder-Only Architecture
Causal self-attention — each token sees only tokens to its left
A decoder-only model applies a causal mask: token at position i can only
attend to positions 1 … i, never to future positions. This makes the model naturally
autoregressive — it can generate sequences token by token. For single-cell biology, "generating"
means sampling a plausible gene expression profile one gene at a time, or predicting what happens
next in a developmental trajectory. Confusingly, scGPT uses a decoder-only architecture despite
its "GPT" name primarily being used for masked pretraining — check the attention mask, not the name.
LEGO analogy: A LEGO builder who is assembling a set step by step, cover page closed.
At step 5, they know what steps 1–4 looked like, but not step 6. They must predict the next brick
from what they've placed so far — exactly how a language model generates the next token.
Model
Biological token
Generation task
Notes
scGPT
Gene name + expression bin
Cell generation, perturbation
Uses condition tokens; despite name, some tasks use bidirectional pretraining
Same motor, four builds. 马达怎么造车 (How the Motor Builds) vs Decoder Variant vs 作用与特点 (Role & Intuition) — scGPT, TranscriptFormer, Cell2Sentence, and Evo 2, side by side.
4
Encoder-Decoder Architecture
Source → Encoded context → Target — cross-attention bridges the two
The encoder-decoder architecture (T5 / BART family) separates two concerns: the encoder reads and
summarizes the source sequence with full bidirectional attention; the decoder generates
the target sequence token-by-token while additionally attending to the encoder's outputs
via cross-attention. This is the natural structure for translation tasks in biology:
translating a control cell state to a perturbed state, translating a DNA sequence to its protein,
or translating a human gene to its mouse ortholog.
LEGO analogy: A LEGO instruction-translator: one expert reads a French instruction
set (encoder — full bidirectional context over the source), then a second expert builds the English
equivalent one step at a time (decoder — causal generation), periodically glancing at the French
expert's summary notes (cross-attention) to stay aligned with the source meaning.
Model
Source
Target
Cross-attention role
T5 / Flan-T5
Corrupted text
Original text
Decoder queries encoder span representations
Enformer (hybrid)
DNA sequence (200kb)
Regulatory track values
Transformer encoder for sequence, prediction head for targets
scELMo (concept)
Control cell profile
Perturbed cell profile
Causal perturbation-conditioned decoding
AlphaFold2 (partial)
MSA + pair features
3D coordinates
Evoformer encodes; structure module decodes into 3D
Bidirectional tabular encoder (intra-/inter-cell attention) feeds a reconstruction decoder — no causal mask anywhere
🔬 Case study: scPRINT-2's additive benchmark
scPRINT-2 (successor to scPRINT-1) is pretrained on 350M cells across 16
organisms and 300+ tissues — currently the largest published scFM training corpus. Its more
interesting contribution, though, is methodological: rather than proposing one more
architecture in isolation, the authors build an additive benchmark — a
"gymnasium" of 42 training runs that switches individual design choices on and off (Post-LN
vs. Pre-LN-style normalization placement, ZiNB vs. MSE reconstruction loss, masked-and-binned
tokens like scGPT vs. continuous log1p expression, contrastive InfoNCE/CCE vs. denoising
objectives) and measures the effect of each swap on denoising, batch correction, cell-type
prediction, and gene-network inference. It's the same question the LEGO wiring comparison
above asks about Post-LN/Pre-LN/Parallel/SwiGLU — just run as a controlled experiment across
an entire model family instead of four hand-picked variants.
The winning combination becomes scPRINT-2: a graph-based encoder paired with a generative
"XPressor" module and decoder, which is what lets an otherwise encoder-flavored scFM also do
expression imputation and counterfactual reasoning ("what would this cell's expression look
like under condition X?") — blurring the encoder-only / encoder-decoder line the same way
AlphaFold2 does above.
Same power unit, tested attachment by attachment. 乐高动力单元如何拼出 scPRINT-2 (How the LEGO Power Unit Builds scPRINT-2) vs scPRINT-2 Module vs 作用与特点 (Role & Intuition) — multi-battery graph input, motor/Transformer, XPressor turbocharger, decoder build-arm, and the 42-configuration gymnasium test rig, side by side.
🧬 Case study: STACK's mask is not GPT's mask
STACK (Dong et al. 2026, bioRxiv, Arc Institute; pretrained on 149M cells from
scBaseCount) is officially an encoder-decoder model, not an encoder-only one.
A learned tokenizer first compresses each cell's expression vector into 100 gene-module
tokens; a tabular transformer encoder then runs intra-cellular attention
(a cell's own tokens) and inter-cellular attention (the same token slot compared across every
cell in the set) — fully bidirectional, no causal mask anywhere — before a reconstruction
decoder maps the resulting latent representations back to expression space.
The part that's easy to conflate: STACK does use masking and does have a decoder, but neither
is the GPT kind. GPT's causal mask is an information-flow constraint — token 3
physically cannot see tokens 4, 5, 6 because they don't exist yet at that generation step.
STACK's rectangular mask is a corruption objective — a block of genes is hidden
across every cell in the set simultaneously, and the model reconstructs it using every other
visible gene in every cell, in both directions, all at once.
STACK rectangular mask
GPT causal mask
Hides
A subset of input genes
Future tokens
Purpose
Reconstruction / representation learning
Next-token generation
Attention
Bidirectional, across cells and modules
Left-to-right only
Decoder type
Reconstruction decoder
Autoregressive decoder
Output
Masked values reconstructed at once
Token 1 → token 2 → token 3 …
LEGO version: the rectangular tarp in the diagram below covers battery
modules 40–60 on every car at once, and the repair crew reconstructs them from
whatever else is visible — that's STACK. GPT is a different kind of restriction entirely:
it's not covering part of a finished car with a tarp, it's forbidding the engineer building
car 3 from looking at cars 4, 5, and 6 because they haven't been built yet. Same word
("mask"), two unrelated mechanisms — one is corruption-and-repair, the other is a build order.
Cars parked together teach each other. 乐高动力单元怎么互相借电 —— STACK (How LEGO Power Units Borrow Power From Each Other) vs STACK Module vs 作用与特点 (Role & Intuition) — learned tokenization, shared cell-set rack, bidirectional intra-/inter-cell encoder attention, masked-reconstruction pretraining, in-context inference, and the Perturb Sapiens catalog, side by side.
5
Positional Encoding
How the model knows "where" each token is — or whether order matters at all
Attention itself is permutation-invariant — shuffle the tokens and the attention
scores change but the mechanism doesn't know they were shuffled. Positional encoding injects
order information. In DNA, position matters enormously (base 500 and base 5000 have different
regulatory contexts). In scRNA-seq, gene order is largely arbitrary — what matters
is the set of expressed genes and their relative levels — so many single-cell models
deliberately omit positional encoding or use gene-rank-based alternatives.
LEGO analogy: Positional encoding is the slot number printed on each
LEGO baseplate. For DNA, the slot number (genomic coordinate) is critical — brick 1000 connects
to brick 1001, not brick 5000. For gene expression, the "slot" is arbitrary — it doesn't matter
whether Gene A is the first or the fifth column in the expression matrix; what matters is its value
relative to the other genes.
Strategy
How it works
Extrapolates?
Best for
Sinusoidal
Fixed sin/cos at each position; added to embeddings
No
Text; original transformer
Learned absolute
Trainable position embedding table
No
BERT (≤512 tokens)
RoPE
Rotate Q,K by position-dependent angle; encodes relative distance in attention score
Partial (NTK scaling)
LLMs, genomic FMs, long sequences
ALiBi
Add linear bias −m·|i−j| to attention score; no learned params
Yes
Long sequences; BLOOMZ, some DNA models
None (set-based)
No position signal; model treats input as unordered set
Same rail, six ways to mark it. 轨道怎么标位置 (How the Rail Marks Position) vs Positional Encoding Variant vs 作用与特点 (Role & Intuition) — Sinusoidal, Learned absolute, RoPE, ALiBi, None (set-based), and Gene-rank, side by side.
Key Models in Biology by Architecture
Single-Cell Genomics
Model
Architecture
Pretraining
Scale
Key use
Geneformer (Theodoris 2023, Nature)
Encoder
MLM on rank-ordered genes
30M cells
Network dosage sensitivity, cell annotation
scBERT
Encoder
MLM on expression bins
1M+ cells
Cell type classification; Performer (linear attn)
scGPT (Wang 2024, Nature Methods)
Decoder
CLM + masked gene modeling
33M cells
Cell generation, perturbation, multi-omics
TranscriptFormer (2025)
Decoder
CLM (gene rank order)
112M cells
Trajectory inference, generation; RoPE
State (Adduri et al. 2025, Arc Institute)
Set Encoder
SE: reconstruction; ST: paired perturbation
167M + 100M cells
Perturbation response; cell-as-token
STACK (Dong et al. 2026, bioRxiv, Arc Institute)
Tabular Encoder-Decoder
Rectangular gene masking across a cell set; bidirectional intra-/inter-cell attention + reconstruction decoder
149M cells (scBaseCount)
In-context counterfactual perturbation/donor prediction; built Perturb Sapiens atlas
Nicheformer (2025, Nat Methods)
Encoder
MLM + spatial masking
Large spatial + scRNA
Spatial transcriptomics; niche-aware embeddings
UCE (Rosen/Roohani et al. 2026, Nature)
Encoder
Masked "was gene expressed?" on ESM2 protein-embedding tokens
Gene modules, regulatory motifs, secondary structure elements
Late layers (8+)
Task-relevant, global context
Cell type identity, pathway activation states, 3D protein contacts
Practical Architecture Selection Guide
1
Define your biological question first
Is the task about understanding (classifying, scoring, representing) or generating (simulating, designing, predicting a new sequence)? Understanding → encoder-only. Generating → decoder-only. Both → encoder-decoder or decoder with conditioning.
2
Does biological order matter for your tokens?
DNA/protein sequences: yes, use positional encoding (RoPE for long sequences). scRNA gene sets: usually no, omit or use gene-rank encoding. Single cells in a population: no — treat as a set (State, STACK model).
3
Check sequence length against O(n²) budget
If your input exceeds ~4,000–8,000 tokens, pure attention becomes expensive. Options: sliding window attention (Longformer style), linear attention (Performer / scBERT), or switch to SSM (HyenaDNA, Evo 2) for very long genomic sequences (>50kb).
4
Choose a pretrained model before training from scratch
For scRNA: start from Geneformer or scGPT. For DNA: DNABERT-2 (<1kb) or Nucleotide Transformer (multi-species). For protein: ESM2 (structure) or ESM3 (multimodal). Training from scratch requires tens of millions of samples and weeks of GPU time — almost always unnecessary for biology.
5
Use the minimal architecture that solves the task
A 6-layer encoder is often sufficient for cell type annotation. A 24-layer decoder is overkill for a 5-class classification task. Larger models require more data to avoid overfitting; smaller fine-tuned models often outperform larger frozen ones on small biological datasets.
Hands-On Resources
Illustrated Transformer
Jay Alammar's visual walkthrough of the encoder-decoder architecture — start here for the block diagram.
