Let's build the GPT Tokenizer
Andrej Karpathy · 5 則 HN 推薦
影片問答
可以請 AI 解釋目前片段，或總結整部影片。如果回答附有時間戳記，點選即可跳到對應片段。
試著問：這裡講的是什麼意思？
需使用 Google 登入。確認前可查看最多會扣除多少點數。
Andrej Karpathy builds the GPT tokenizer from scratch, implementing byte pair encoding along with encode and decode. He shows how many odd LLM behaviors trace back to tokenization, and covers Unicode byte encodings along the way.
HN 推薦的原因： Recommended as the best deep dive on tokenizers, worth the two hours for anyone working with LLMs.
“Worth spending 2h for the deeper understanding if you deal with LLMs.” 閱讀 HN 討論 ↗