Let's build the GPT Tokenizer
Andrej Karpathy · 5 HN recommendations
Video Q&A
Ask AI to explain this scene or summarize the whole video. When an answer includes timestamps, select one to jump to that moment.
Try asking: What is the speaker explaining here?
Google sign-in is required. Review the maximum credit cost before confirming.
Andrej Karpathy builds the GPT tokenizer from scratch, implementing byte pair encoding along with encode and decode. He shows how many odd LLM behaviors trace back to tokenization, and covers Unicode byte encodings along the way.
Why HN recommends it: Recommended as the best deep dive on tokenizers, worth the two hours for anyone working with LLMs.
“Worth spending 2h for the deeper understanding if you deal with LLMs.” Read on HN ↗