RESEARCH PROTOTYPE · TOKENIZATION · CONTROLLED ABLATION
Tokenizer Ablation
How much does tokenizer choice change what a small model learns? This experiment trains a byte-level BPE tokenizer from scratch, sweeps its vocabulary size, and measures code-identifier classification accuracy with the same fixed-width classifier at fixed token and byte budgets.
Measure the tokenizer, not a moving target
- Pure-Python byte-level BPE training with byte fallback.
- Vocabulary sweep: 1k, 4k, 16k, and 32k.
- Eight-class code-identifier classification task.
- Compare accuracy and classifier fit time at fixed token and UTF-8 byte budgets.
- Optional tiktoken cl100k_base and Hugging Face BERT WordPiece references.
The corpus and task are generated and intentionally limited. The results are not claims about pretrained LLMs, arithmetic, or language-model quality.
Run the experiment
cd tokenizer-ablation
python -m venv .venv
source .venv/bin/activate
pip install -e ".[dev]"
tokenizer-ablation --output results/ablation.json
pytest