RESEARCH PROTOTYPE · SEQUENCE MODELS · CONTROLLED COMPARISON
RNN vs LSTM vs GRU vs Tiny Transformer
At small scale, does attention beat recurrence—or mainly change the compute bill? This experiment approximately matches model parameter counts, compares character-level language modeling, and probes recall across 100 distractor steps.
One budget, four architectures
- Vanilla RNN, LSTM, and GRU against a two-layer, four-head causal Transformer.
- Validation loss vs training steps and perplexity.
- 200-character samples and per-epoch training time.
- Autoregressive inference tokens/second and latency.
- Synthetic long-range recall with a 50% random baseline.
The bundled corpus and recall probe are intentionally small. Results depend on training budget and hardware; this is a reproducible experiment, not a universal architecture ranking.
Run the comparison
cd sequence-models
python -m venv .venv
source .venv/bin/activate
pip install -e ".[dev]"
sequence-models
pytest