Files

02_bonus_bytepair-encoder

Name		Name	Last commit message	Last commit date
parent directory ..
gpt2_model		gpt2_model
README.md		README.md
bpe_openai_gpt2.py		bpe_openai_gpt2.py
compare-bpe-tiktoken.ipynb		compare-bpe-tiktoken.ipynb
requirements-extra.txt		requirements-extra.txt

README.md

Chapter 2: Working with Text Data

compare-bpe-tiktoken.ipynb benchmarks various byte pair encoding implementations
bpe_openai_gpt2.py is the original bytepair encoder code used by OpenAI