Let's build the GPT Tokenizer
Andrej Karpathy · 5 则 HN 推荐
视频问答
可以让 AI 解释当前片段，或总结整部视频。如果回答附有时间戳，点击即可跳到对应片段。
试着问：这里讲的是什么意思？
需要用 Google 登录。确认前可查看最多会扣除多少积分。
Andrej Karpathy builds the GPT tokenizer from scratch, implementing byte pair encoding along with encode and decode. He shows how many odd LLM behaviors trace back to tokenization, and covers Unicode byte encodings along the way.
HN 推荐的原因： Recommended as the best deep dive on tokenizers, worth the two hours for anyone working with LLMs.
“Worth spending 2h for the deeper understanding if you deal with LLMs.” 阅读 HN 讨论 ↗