youtube.nixfred.com nixfred.com
Andrej Karpathy AI Deep Learning

Let's build the GPT Tokenizer

Karpathy builds a byte pair encoding tokenizer from scratch and argues that tokenization is the root of a startling amount of LLM weirdness: bad spelling, failed string reversal, worse performance in non English languages, arithmetic errors, the Python indentation problem in GPT-2, and the unspeakable SolidGoldMagikarp tokens. It starts at Unicode and UTF-8, works up through the BPE merge algorithm, the GPT-2 and GPT-4 splitting patterns, special tokens, and SentencePiece, then closes on how to choose a vocabulary size.

Published Feb 20, 2024 2:13:34 video 105 min read Added Jul 30, 2026 Open on YouTube →

At a glance

Andrej Karpathy opens with a set face, and says why: tokenization is his least favorite part of working with large language models. It is necessary anyway, because it is hairy and gnarly and full of hidden foot guns, and because a startling share of the oddness people blame on model architecture traces straight back to it. "A lot of the issues that may look like just issues with the neural network architecture or the large language model itself are actually issues with the tokenization, and fundamentally trace back to it."

Over two hours and thirteen minutes he builds a byte pair encoding tokenizer from an empty cell: Unicode code points, UTF-8 bytes, a pair counter, a merge function, a training loop, then encode and decode, including the specific bugs each one hides. Then he adds everything production tokenizers put on top: the regex that pre splits text so merges can never cross category boundaries, read piece by piece; what GPT-4 changed in that regex; special tokens and the model surgery they require; SentencePiece and the real configuration differences against tiktoken; and how vocabulary size trades off against the embedding table and the softmax. The reference implementation is minbpe, published alongside the lecture with a four step exercise that walks you to your own GPT-4 tokenizer.

The last twenty minutes are the payoff. He returns to the list of crimes he opened with and explains each one with the mechanism now visible: why GPT-4 miscounts the letters in .DefaultCellStyle, why it cannot reverse that string but can if you first make it list the characters out, why "hello how are you" is 5 tokens and its Korean translation is 15, why 677 arrives as two tokens, why GPT-2 wrote bad Python, why a trailing space in your prompt pushes the model out of distribution, and why asking GPT-2 about a Reddit user called SolidGoldMagikarp makes it insult you.

His own closing position: do not brush this off, there are foot guns and security issues and AI safety issues living in this stage. And eternal glory goes to anyone who manages to get rid of it.

The deep explanation

Where the last video left off, and why that tokenizer was a toy

Karpathy starts by rewinding to Let's build GPT from scratch, the previous video in the series, because that video already did tokenization. It just did a very naive, simple version of it.

In that Colab notebook the training set was the tiny Shakespeare dataset, which in the beginning is just a large string in Python. Just text. The question the whole field has to answer is how you plug text into a neural network, and the answer there was the simplest possible one: collect every distinct character that occurs in the string, find there are 65 of them, and build a lookup table from each one character string to an integer. Tokenize "hi there" and you get back a short sequence of integers. Take the first 1,000 characters of the dataset and encode them, and because the scheme is character level, you get exactly 1,000 tokens out.

Those integers then index an embedding table. Sixty five possible tokens means an embedding table with 65 rows. You take the integer for a token, use it as a lookup into that table, pluck out the corresponding row, and that row is a vector of trainable parameters that you train by back propagation. That vector is what feeds into the Transformer. The Transformer never sees a character. It sees a row of a table.

That is the whole mechanism, and it survives intact into state of the art models. What changes is the vocabulary. In practice nobody works at the character level. They work at the chunk level, and the chunks are constructed by algorithms like byte pair encoding, which is the subject of the next two hours.

For a paper that introduced byte level BPE as the tokenization mechanism for large language models, Karpathy points at the GPT-2 paper, Language Models are Unsupervised Multitask Learners. Scroll to the section on input representation and you get the properties they wanted from a tokenizer and the conclusion they landed on: a vocabulary of 50,257 possible tokens, with a context size of 1,024 tokens, up from GPT-1's 512. In the attention layer of the Transformer every token attends to the previous tokens in the sequence, and it can see up to 1,024 of them.

Tokens, in other words, are the atom. Everything is in units of tokens. "Tokenization is the process for translating strings or text into sequences of tokens, and vice versa." Search the Llama 2 paper for the word "token" and you get 63 hits, including the headline number that they trained on two trillion tokens of data.

The crime sheet he opens with

Before any code, Karpathy front loads the motivation with a list of failures, because he wants the reader sufficiently motivated for why all of this gross work matters. Every item on this list comes back at the 1:51:41 mark with a mechanism attached:

His framing: "tokenization is at the heart of a lot of weirdness in large language models, and I would advise that you do not brush it off."

Tokenization by example, before any code

Rather than start in a notebook, he starts in a browser, at Tiktokenizer. What he likes about this web app is that tokenization runs live in your browser in JavaScript, so you type on the left and watch the chunks appear on the right, each token in its own color. The tokenizer selected at the start is gpt2, and the sample string he has pasted in currently tokenizes into 300 tokens.

Walk through the English sentence first. The word "Tokenization" becomes two tokens, 30642 and 1634. The token " is" is 318. The token " at" is 379. The token " the" is 262. Turn on the whitespace display at the bottom of the app and you can see there are spaces and newline characters in there, though you can hide them for clarity. The thing to notice is that the space is part of the token chunk. The common token is " the" with its leading space attached, not "the".

Then arithmetic, and the arbitrariness begins. In 127 + 677, the 127 feeds into the model as a single token, but 677 feeds in as two separate tokens, " 6" and "77". So the large language model has to take account of that split and somehow process it correctly inside its network. He scrolls through four digit numbers and the pattern is pure chance: 804 breaks into " 8" and "04", 1275 into "12" and "75", 6773 into " 6" and "773", 8041 into " 8" and "041". "Sometimes you have multiple digits as a single token, sometimes you have individual digits as many tokens, and it's all kind of pretty arbitrary and coming out of the tokenizer."

Then the egg demonstration, which is the cleanest illustration of how little the tokenizer cares about concepts. The string "Egg" at the start of a sentence becomes two tokens. Write "I have an egg" and " egg" with its leading space is suddenly a single token, for the exact same letters. Lowercase "egg" by itself is a single token too, and you can see from the color that it is a different token, because the tokenizer is case sensitive. Capital "EGG" would be different tokens again. So for one concept, egg, whether it sits at the beginning of a sentence or in the middle, lowercase or uppercase or mixed, you get a handful of completely different token ids, and the language model has to learn from raw internet text that all of those are the same thing and group them in its parameters. "It has to sort of group them in the parameters of the neural network and understand, just based on the data patterns, that these are all very similar."

Then Korean. He pastes in an introduction from OpenAI's ChatGPT written in Korean, because non English languages work measurably worse in ChatGPT and part of that is the tokenizer, not just the model. When you train the tokenizer there is a training set, and there is a lot more English in it than anything else, so the tokenizer learns a lot more long tokens for English. Take a single English sentence and it might be 10 tokens. Translate it into Korean or Japanese and the count goes up substantially, because the chunks over there are a lot more broken up. Three separate taxes follow from that. The sequence length of every document bloats, so in the attention of the Transformer those tokens run out of context sooner. All the non English text is stretched out from the Transformer's point of view. And translating it back to English would be significantly fewer tokens for the identical meaning.

Then Python, which is the single most vivid frame in the first fifteen minutes. He pastes in a snippet of Python doing FizzBuzz and points at the indentation. Every individual space is its own token. Token 220, over and over: 220, 220, 220, 220. Then " if" as one token. So when the Transformer consumes that text, all those spaces feed in one by one through the whole sequence. "This is being extremely wasteful, tokenizing it in this way." And as a result GPT-2 is not very good with Python, and it has nothing to do with coding or the language model itself. It is that if you indent with spaces in Python the way everybody does, you bloat the text out and shred it across far too much of the sequence, and you run out of context length.

Then the fix, live. He scrolls up and switches the tokenizer from gpt2 to cl100k_base, which is the GPT-4 tokenizer. The same string that was 300 tokens drops to 185 tokens. Roughly half, and roughly because the GPT-4 vocabulary is roughly double the GPT-2 one: about 50,000 to about 100,000. That is a good thing, because the same text is now squished into half as many tokens, which is a denser input, and because every token attends to a finite number of tokens before it, you are now able to see roughly twice as much text as context when predicting what comes next.

But not infinitely good, and he flags the trade off here before coming back to it an hour and a half later. As you increase the number of tokens, your embedding table gets a lot larger, and at the output you are trying to predict the next token, so the softmax grows as well. "There's some kind of a sweet spot somewhere, where you have a just right number of tokens in your vocabulary, where everything is appropriately dense and still fairly efficient."

The specific GPT-4 improvement he wants noticed is whitespace. In the same Python snippet, four spaces are now represented as one single token. Then a token for the next three spaces, then " if". Seven spaces elsewhere get grouped into a single token. "This was a deliberate choice made by OpenAI when they designed the GPT-4 tokenizer." What it does is densify Python, so the model can attend to more code before it when predicting the next token. Which means the improvement in Python coding ability from GPT-2 to GPT-4 is not purely a matter of the language model, the architecture and the optimization. A real part of it is the design of the tokenizer and how it groups characters into tokens.

Strings in Python are sequences of Unicode code points

Now to code. The goal restated: take strings, turn them into integers in some fixed vocabulary, use those integers to look up vectors in a table, and feed those vectors into the Transformer. The reason it gets tricky is that we do not just want the simple English alphabet. We want Korean, where 안녕하세요 is hello. We want every special character you might find on the internet, emoji included.

So what is a string in Python anyway? Go to the documentation and strings are immutable sequences of Unicode code points.

What is a Unicode code point? It is defined by the Unicode Consortium as part of the Unicode standard, which is a definition of roughly 150,000 characters across 161 scripts, what they look like, and which integers represent them. The standard is very much alive: the latest version at recording time is 15.1, from September 2023.

You access the code point of a single character in Python with ord. ord("h") is 104. It can get arbitrarily larger: the emoji he tries comes back as 128,000 and a Korean character comes back around 50,000. You cannot pass a whole string, because ord takes a single character and returns its integer. But you can map it across a string, [ord(x) for x in "some string"], and get the full list of code points.

So why not just use the code points and skip tokenization entirely

A fair question, and he asks it directly: we already have integers here, so why not use them natively as the vocabulary and have no tokenization at all?

Two reasons. First, the vocabulary would be quite long: 150,000 different code points. Second, and more worryingly, the Unicode standard is alive and keeps changing, so it is not a stable representation to build a model on. A vocabulary you have to revise when the Consortium ships a new version is not a vocabulary.

Unicode byte encodings: UTF-8, UTF-16, UTF-32

Something better means encodings. The Unicode Consortium defines three: UTF-8, UTF-16 and UTF-32. These are the ways you take Unicode text and translate it into binary data, into byte streams.

UTF-8 is by far the most common. It takes every single code point and translates it to a byte stream of between one and four bytes, so it is a variable length encoding: depending on the code point, you end up with one to four bytes according to the schema. UTF-32 is nice in that it is fixed length rather than variable length, but it has many other downsides. The full spectrum of pros and cons across the three is out of scope, but he points at a blog post he enjoyed, A Programmer's Introduction to Unicode, whose reference list at the end includes the UTF-8 Everywhere Manifesto. That manifesto makes the case for why UTF-8 is significantly preferred and a lot nicer than the alternatives, and why it dominates the internet. The one advantage he calls out to give you a sense of it: UTF-8 is the only one of the three that is backwards compatible with the much simpler ASCII encoding of text.

Then he does it in the notebook. Python strings have .encode(), so "...".encode("utf-8") gives you a bytes object, which prints badly, so he wraps it in list() to get the raw bytes as integers. Those are the actual bytes representing the string under UTF-8.

Run the same string through UTF-16 and you get a slightly different byte stream, and the disadvantage shows up immediately: zero, something, zero, something, zero, something. For simple ASCII or English characters you get that wasteful alternating structure throughout. UTF-32 is worse in the same way, a lot of zeros followed by something. "So this is not desirable."

So UTF-8 it is. But if we use UTF-8 naively, these are byte streams, which implies a vocabulary of only 256 possible tokens. And that is very, very small. What it would do is stretch all of our text out over very, very long sequences of bytes. The embedding table would be tiny and the final prediction layer would be tiny, which is fine, but the sequences would be enormous, and we have a finite context length and finite attention budget for computational reasons. So we would never attend to enough text before us to predict the next token well.

The requirement is now precise. We want to keep the UTF-8 encoding of strings, and we want a vocabulary size larger than 256 that we can tune as a hyperparameter. The answer is byte pair encoding, which compresses those byte sequences by a variable amount.

The daydream: deleting tokenization

Before building it, he takes a detour he clearly wishes were the main road. "I would love nothing more than to be able to feed raw byte sequences into language models."

There is a paper on how that could potentially be done, from the summer before recording: MEGABYTE: Predicting Million-byte Sequences with Multiscale Transformers. The problem is that you have to go in and modify the Transformer architecture, because attention becomes extremely expensive when the sequences are that long. So they propose a hierarchical structuring of the Transformer that would let you feed in raw bytes. Their conclusion, which he reads out: "Together, these results establish the viability of tokenization free autoregressive sequence modeling at scale."

"Tokenization free would indeed be amazing. We would just feed byte streams directly into our models." But he does not believe it has been proven out yet by enough groups at enough scale. He hopes someone comes up with it. For now we come back and compress with BPE.

The byte pair encoding algorithm, by hand

The Wikipedia page is "actually quite instructive as far as the basic idea goes," so he walks its example rather than inventing one. The toy vocabulary has only four elements, a, b, c and d, and the input sequence is aaabdaaabac, which is eleven symbols long. Too long, and we would like to compress it.

The loop is three steps:

  1. Iteratively find the pair of tokens that occurs most frequently.
  2. Replace every occurrence of that pair with a single new token, appended to the vocabulary.
  3. Repeat.

Run it. The pair aa occurs most often, so we mint a new token, call it Z, and replace every occurrence of aa with Z. A sequence of 11 characters with a vocabulary of 4 is now a sequence of 9 tokens with a vocabulary of 5, because we created a fifth element standing for the concatenation of aa.

Repeat. Now the most frequent pair is ab, so we mint Y, and Y stands for ab. Seven tokens, vocabulary of six.

Repeat one more time. The most frequent pair is now ZY, which is a pair of minted tokens, so the merge is on top of previous merges. Mint X for ZY. Five tokens, vocabulary of seven. We went from 11 tokens over a vocabulary of 4 to 5 tokens over a vocabulary of 7, and that is the entire trade.

input: a a a b d a a a b a c a a a b d a a a b a c 11 tokens, vocabulary 4 most frequent pair (a, a) merge 1: aa becomes Z Z a b d Z a b a c 9 tokens, vocabulary 5 most frequent pair (a, b) merge 2: ab becomes Y Z Y d Z Y a c 7 tokens, vocabulary 6 most frequent pair (Z, Y) merge 3: ZY becomes X, a merge of two minted tokens X d X a c 5 tokens, vocabulary 7 nothing left worth merging Every box is as wide as the original text it covers, so all four rows span the same eleven characters.
Figure 1. The three merges Karpathy walks at 23:50, on the toy string aaabdaaabac that opens minbpe's quick start. Sequence length falls 11, 9, 7, 5 while vocabulary size rises 4, 5, 6, 7. Minted tokens are themselves eligible for merging, which is why the third merge is a pair of merges, and why the result is not a tree.

The real thing works exactly the same way with a different starting point. "In the exact same way, we start out with byte sequences, so we have 256 vocabulary size, but we're now going to go through these and find the byte pairs that occur the most, and we're going to iteratively start minting new tokens, appending them to our vocabulary and replacing things." The output is two things: a compressed training dataset, and an algorithm for encoding any arbitrary sequence into that vocabulary and decoding it back to strings.

Starting the implementation: 533 code points, 616 bytes

He takes the first paragraph of a blog post he enjoyed, pastes it into the notebook as one very long line, and encodes it to UTF-8. The tokens at this point are a raw stream of bytes, and to make them easier to manipulate in Python he converts every byte to an integer and wraps the whole thing in a list.

The numbers matter here because they make the UTF-8 expansion concrete. The original paragraph is 533 code points long. Encoded to UTF-8 it is 616 bytes, so 616 tokens at this stage. The reason it grew is exactly the variable length encoding: simple ASCII characters become a single byte each, but the more complex Unicode characters in the paragraph become multiple bytes, up to four.

get_stats: counting every consecutive pair

First step of the algorithm: find the pair of bytes that occurs most frequently, because that is the pair we are going to merge. He stops and tells the reader to write the function themselves if they are following along in a notebook, then pastes his own.

He calls it get_stats. It takes a list of integers, uses a dictionary to keep the counts, iterates consecutive elements of the list with the Pythonic zip(ids, ids[1:]) trick from the previous video, and increments by one for every pair it sees. The keys come out as tuples of consecutive elements and the values are counts.

To read the result he sorts it in a slightly compound way worth pausing on: iterate stats.items(), which returns key value pairs, build a list of (value, key) instead so that Python's default sort uses the count first, and reverse it so it descends.

Top of the list: (101, 32), which occurred 20 times. He double checks by searching the token list for 101, 32 and finds exactly 20 occurrences. Then he asks what the pair actually is, using chr, the opposite of ord. chr(101) is e and chr(32) is a space. So the single most common byte pair in this English paragraph is "e followed by a space," which means a great many words in it end in the letter e.

merge: replacing the pair, and the bound check that bites

Now mint a new token. The existing tokens run from 0 to 255, so the new one gets id 256, and we iterate the whole list swapping every 101, 32 for 256.

His merge function takes a list of ids, the pair to replace, and the new index idx. It builds a new list, walks the old one left to right, and at each position checks whether the current element and the next one match the pair.

And here is the careful bit, the thing he flags explicitly: "here is a bit of a tricky condition that you have to append if you're trying to be careful, and that is that you don't want this here to be out of bounds at the very last position." If you are standing on the rightmost element of the list and you look at ids[i+1], you get an index error. So the condition has to also assert you are not at the very last element. On a match you append the replacement index and advance the position by two, skipping the whole pair. Otherwise you copy the element at that position and advance by one.

He tests it on a toy case first: the list [5, 6, 6, 7, 9, 1] with the pair (6, 7) replaced by 99 gives back the list with the 6 and the 7 collapsed into a single 99. Then he runs it for real, merging the top pair into 256.

The verification is clean. The list was length 616. It is now length 596, a decrease of exactly 20, which makes sense because there were 20 occurrences. Search the new list for 256 and you find plenty of them. Search it for 101, 32 and there are none, where the original had plenty. One pair successfully merged.

The training loop: 20 merges, a forest rather than a tree, and 1.27x

Before the loop he swaps the training text for something bigger: instead of the first paragraph he takes the entire blog post, stretched out into a single line, because longer text gives more representative statistics for the byte pairs and a more sensible result.

Then the loop. First decide the final vocabulary size, which is the hyperparameter. He picks 276, because 256 raw byte tokens plus 20 merges is 276, and 20 merges is a comfortable number to look at. He copies the token list with list(tokens) and creates a merges dictionary that will hold the mapping from (child one, child two) to the new token id.

A nice aside on what that dictionary describes. "What we're going to be building up here is a binary tree of merges, but actually it's not exactly a tree, because a tree would have a single root node with a bunch of leaves. For us we're starting with the leaves on the bottom, which are the individual bytes, those are the starting 256 tokens, and then we're starting to merge two of them at a time. And so it's not a tree, it's more like a forest."

Twenty iterations: find the most common pair, mint the next integer starting at 256, print the merge, replace every occurrence, record the pair in merges. The first merge is the one we already did by hand, (101, 32) becoming 256.

Two properties of that output he wants understood. The individual tokens 101 and 32 can still occur in the sequence after the merge. It is only when they occur exactly consecutively that they become 256. And the newly minted token 256 is itself eligible for merging: the twentieth merge in his run consumes token 259, which was itself produced by an earlier merge, to produce token 275. "Every time we replace these tokens they become eligible for merging in the next round of iteration. So that's why we're building up a small sort of binary forest instead of a single individual tree."

Then the compression ratio. He started with 24,000 bytes and after 20 merges has about 19,000 tokens. Divide the two and the compression ratio is roughly 1.27x, achieved with 20 merges. More vocabulary elements, more compression.

The tokenizer is a completely separate stage from the model

This is the point he stops and draws a diagram for, because it is the thing people blur most often.

"The tokenizer is a completely separate object from the large language model itself. Everything in this lecture, we're not really touching the LLM itself. We're just training the tokenizer. This is a completely separate pre processing stage."

What that means concretely:

And because the tokenizer's training set is its own choice, the mixture in it is a real design decision with a real downstream effect. "If you add some amount of data, like say you have a ton of Japanese data in your tokenizer training set, then that means that more Japanese tokens will get merged, and therefore Japanese will have shorter sequences, and that's going to be beneficial for the large language model, which has a finite context length." The same logic applies to code against prose. How much of a language is in the tokenizer's corpus determines the density that language gets in token space.

Hold onto that, because it is also the entire explanation of SolidGoldMagikarp two hours later.

STAGE 1 train the tokenizer tokenizer training set its own documents byte pair encoding run once, offline merges + vocab the entire tokenizer loaded by raw text Unicode code points tokenizer a translation layer token sequence integers encode encode decode decode STAGE 2 train the model LLM training set often a different set tokenize all of it one massive pass token stream on disk raw text thrown away Transformer sees only integers Two stages, two training sets. When they disagree, you get tokens the model has never been trained on.
Figure 2. The diagram Karpathy stops to draw at 39:20. The tokenizer is a one time pre processing stage with its own corpus; the model reads integers off disk and never touches text. The gap between the two training sets is the whole explanation of SolidGoldMagikarp at 2:04:28.

decode: tokens back to text, and the invalid start byte

With merges trained, both directions become implementable. He starts with decode: given a list of integers, return a Python string.

His solution builds a preprocessing variable called vocab, a dictionary mapping token id to the bytes object for that token. The first 256 entries are the raw bytes for ids 0 through 255. Then he walks the merges in order and populates the rest by concatenating the bytes of the first child with the bytes of the second child. These are bytes objects, so the + here is bytes concatenation.

One tricky thing he flags: he iterates the dictionary with .items(), and it genuinely matters that this runs in the order the items were inserted into merges. "Luckily, starting with Python 3.7 this is guaranteed to be the case, but before Python 3.7 this iteration may have been out of order with respect to how we inserted elements into merges, and this may not have worked."

Then the function itself: iterate the ids, look up each one's bytes in vocab, join them all with b"".join(...) to get the raw byte string, and call .decode("utf-8") on it. Where encoding went from string to bytes, this goes from bytes back to string.

And then the bug, which he deliberately shipped so he could show it. Decode [97] and you get a, nothing crazy. Decode [128] as a single element and Python raises:

UnicodeDecodeError: 'utf-8' codec can't decode byte 0x80 in position 0: invalid start byte

To understand that you go back to the UTF-8 page. Multi byte characters have to carry a specific envelope: the leading byte announces how many bytes follow with a prefix of ones, and the continuation bytes each start 10. The binary representation of 128 is a one followed by all zeros, and that fits none of the rules. A leading 1 must be followed by another 1 and then a 0, and then the content. So 128 standing alone is an invalid start byte and cannot be decoded.

The fix is the errors argument of bytes.decode. The default is errors="strict", which raises. Switch it to errors="replace" and you get back the Unicode replacement character instead of an exception.

Why this matters in production: "not every single byte sequence is valid UTF-8, and if it happens that your large language model, for example, predicts your tokens in a bad manner, then they might not fall into valid UTF-8, and then we won't be able to decode them." So the standard practice is errors="replace", which is also what you find in the code OpenAI released. And the operational tell for you as a user: whenever you see that replacement character in an output, "something went wrong and the LLM output was not a valid sequence of tokens."

encode: text to tokens, and the lowest index merge

Now the other direction. Given a string, produce a list of token integers.

Start the same way: encode the text to UTF-8, call list() on the bytes to get a list of integer bytes. Those are the starting tokens, the raw bytes of the sequence. But some of them need merging, according to the merges dictionary.

And the order is not optional. "Merges was built from top to bottom, and this is sort of the order in which we inserted stuff into merges, and so we prefer to do all these merges in the beginning before we do these merges later, because for example this merge over here relies on the 256 which got merged here." Later merges are defined in terms of earlier ones, so you must apply them in the order they were learned.

The loop reuses get_stats from training, but for a different purpose. "At this point we don't actually care how many times they occur in the sequence, we only care what the raw pairs are in that sequence." He uses only the dictionary's keys, as the set of candidate merges present in the text.

Then the selection, which is the one genuinely clever line in the implementation. He wants the pair in stats that has the lowest index in merges, because early merges must happen before late ones. He does it with min over the dictionary's keys with a key function that looks each pair up in merges and returns its index:

pair = min(stats, key=lambda p: merges.get(p, float("inf")))

The float("inf") fallback is the part worth dwelling on. If a pair in the token sequence is not a merging pair at all, it has no index in merges, so it is not eligible, and infinity guarantees it loses every comparison and never gets selected. "The reason infinity is nice here is because for sure we're guaranteed that it's not going to participate in the list of candidates when we do the min."

But that leaves a failure mode he catches out loud. If there is nothing left to merge, every candidate evaluates to infinity, and min returns whichever pair happens to come first in stats, arbitrarily. That pair is not actually mergeable. So the check after the min is a test for exactly that: "if this pair is not in merges that was returned, then this is a signal for us that actually there was nothing to merge, no single pair can be merged anymore." In that case, break.

If the pair is in merges, look up its index and call merge(tokens, pair, idx), which returns a new list with every occurrence replaced. Loop until nothing can be merged, then return the tokens. He checks the output and spots 32 in there, which is a space in ASCII, so it looks right.

Then he comes back for the edge case he left out. "This is not quite the right implementation just yet, because we are leaving out a special case." If the input is a single character or an empty string, stats is empty, and that blows up inside min. So the guard is a length check: run the loop only while len(tokens) >= 2, because with fewer than two tokens there is nothing to merge and you can just return.

The round trip tests, and the one direction that does not hold

Two test cases, and the second one is the interesting one.

Take a string, encode it, decode it back, and you expect the same string. Is that true for all strings? In his tests it holds, and he believes it holds in general. He checks it on the training text the tokenizer was trained on, and then on a piece of validation text he grabbed from a web page the tokenizer has never seen, and both round trip correctly. "That gives us some confidence that this was correctly implemented."

But the reverse direction is not an identity, and he is explicit about why: "going backwards is not, you're not going to have an identity going backwards, because as I mentioned, not all token sequences are valid UTF-8 byte streams, and so therefore some of them can't even be decodable." Decode then encode is not guaranteed to give you back your tokens. Encode then decode is.

That closes out the basic tokenizer. "The parameters of this tokenizer really are just this dictionary of merges, and that basically creates the little binary forest on top of raw bytes." With the merges table you can encode and decode between raw text and token sequences, and that is the complete object.

"What we're going to do now though is we're going to look at some of the state of the art large language models and the kinds of tokenizers that they use, and we're going to see that this picture complexifies very quickly."

The complete tokenizer, as code

Four functions and two dictionaries. This is the whole object, in the shape minbpe ships it, with his own comments kept where they explain a decision rather than a line.

The two helpers, which training and encoding both use:

def get_stats(ids, counts=None):
    """
    Given a list of integers, return a dictionary of counts of consecutive pairs
    Example: [1, 2, 3, 1, 2] -> {(1, 2): 2, (2, 3): 1, (3, 1): 1}
    """
    counts = {} if counts is None else counts
    for pair in zip(ids, ids[1:]):  # iterate consecutive elements
        counts[pair] = counts.get(pair, 0) + 1
    return counts


def merge(ids, pair, idx):
    """
    In the list of integers (ids), replace all consecutive occurrences
    of pair with the new integer token idx
    Example: ids=[1, 2, 3, 1, 2], pair=(1, 2), idx=4 -> [4, 3, 4]
    """
    newids = []
    i = 0
    while i < len(ids):
        # if not at the very last position AND the pair matches, replace it
        if ids[i] == pair[0] and i < len(ids) - 1 and ids[i+1] == pair[1]:
            newids.append(idx)
            i += 2
        else:
            newids.append(ids[i])
            i += 1
    return newids

The i < len(ids) - 1 clause in merge is the out of bounds guard he stops to explain. Without it the last element of the list raises an index error, and only on inputs whose final element happens to be the first half of the pair, which is exactly the kind of bug that survives a casual test.

Training, which is the loop from the walkthrough with the bookkeeping added:

def train(self, text, vocab_size, verbose=False):
    assert vocab_size >= 256
    num_merges = vocab_size - 256

    # input text preprocessing
    text_bytes = text.encode("utf-8")  # raw bytes
    ids = list(text_bytes)             # list of integers in range 0..255

    # iteratively merge the most common pairs to create new tokens
    merges = {}                                            # (int, int) -> int
    vocab = {idx: bytes([idx]) for idx in range(256)}      # int -> bytes
    for i in range(num_merges):
        stats = get_stats(ids)              # count every consecutive pair
        pair = max(stats, key=stats.get)    # the pair with the highest count
        idx = 256 + i                       # mint the next available id
        ids = merge(ids, pair, idx)         # replace every occurrence
        merges[pair] = idx                  # save the merge
        vocab[idx] = vocab[pair[0]] + vocab[pair[1]]
        if verbose:
            print(f"merge {i+1}/{num_merges}: {pair} -> {idx} "
                  f"({vocab[idx]}) had {stats[pair]} occurrences")

    self.merges = merges   # used in encode()
    self.vocab = vocab     # used in decode()

Three things worth reading closely. num_merges = vocab_size - 256 is the arithmetic behind his choice of 276: twenty merges on top of the byte floor, which is also the arithmetic behind GPT-2's 50,257 with its 50,000 merges and one special token. max(stats, key=stats.get) is the idiom for "the key with the largest value," which is how the most frequent pair gets selected. And vocab[idx] = vocab[pair[0]] + vocab[pair[1]] is bytes concatenation, built up in merge order, which is the decoding table assembled for free during training rather than reconstructed afterwards.

Decoding, three lines:

def decode(self, ids):
    # given ids (list of integers), return Python string
    text_bytes = b"".join(self.vocab[idx] for idx in ids)
    text = text_bytes.decode("utf-8", errors="replace")
    return text

That errors="replace" is the fix for the invalid start byte. The default, errors="strict", raises on any byte sequence a language model emits that does not happen to be valid UTF-8, which is a crash in production triggered by a sampling accident.

Encoding, with the subtlety he spends several minutes on preserved in the comment:

def encode(self, text):
    # given a string text, return the token ids
    text_bytes = text.encode("utf-8")  # raw bytes
    ids = list(text_bytes)             # list of integers in range 0..255
    while len(ids) >= 2:
        # find the pair with the lowest merge index
        stats = get_stats(ids)
        pair = min(stats, key=lambda p: self.merges.get(p, float("inf")))
        # subtle: if there are no more merges available, the key will
        # result in an inf for every single pair, and the min will be
        # just the first pair in the list, arbitrarily
        # we can detect this terminating case by a membership check
        if pair not in self.merges:
            break  # nothing else can be merged anymore
        # otherwise let's merge the best pair (lowest merge index)
        idx = self.merges[pair]
        ids = merge(ids, pair, idx)
    return ids

The while len(ids) >= 2 is the empty string and single character guard. The float("inf") default is what makes an ineligible pair lose every comparison. And the membership check after the min is the one piece of this function you would never write unless you had hit the bug, because the failure is silent: min happily returns a pair that cannot be merged, and without the check you would merge it with whatever merges returns next.

Note what is not in here. No regex splitting, so this tokenizer will merge e with a following space and will produce dog. as a token given the chance. No special tokens. Those are the next two sections, and in minbpe they are the difference between BasicTokenizer and RegexTokenizer.

The regex that stops the merges nobody wants

Back to the GPT-2 paper, 2019, about five years before the recording, and specifically its input representation section. Everything in it matches what we just built, right up to one point, and then it departs.

The problem they describe is punctuation. Take a common word like dog. It occurs very frequently, and it occurs right next to all kinds of punctuation: dog., dog!, dog? and so on. Naively, BPE will happily merge all of those into single tokens, and you end up with a cluster of near duplicate tokens that are just "dog with a slightly different punctuation mark." As Karpathy puts it, "it feels like you're clustering things that shouldn't be clustered, you're combining kind of semantics with punctuation, and this feels suboptimal." The paper says their experiments agreed.

So what they do is enforce, top down and manually, that some types of characters should never be merged together, as a set of rules layered on top of byte pair encoding. The place that happens is encoder.py in the GPT-2 repository. Karpathy registers a complaint about the name in passing: "I don't personally love that they call it encoder.py, because this is the tokenizer, and the tokenizer can do both encode and decode, so it feels kind of awkward to me."

The core of it is one regex pattern that looks very complicated, and one import that is easy to miss. The file does not import re, it does import regex as re, where regex is a third party package you pip install that extends the standard library's re with more power, including the Unicode property classes this pattern depends on.

Here is the pattern as released:

self.pat = re.compile(r"""'s|'t|'re|'ve|'m|'ll|'d| ?\p{L}+| ?\p{N}+| ?[^\s\p{L}\p{N}]+|\s+(?!\S)|\s+""")

He copies it into the notebook and takes it for a spin with re.findall(pat, text), exactly as their code does. findall walks the string left to right, tries to match the pattern wherever it currently stands, and collects every occurrence into a list.

The first structural thing to see is that the pattern is "made up of a lot of ors," the vertical bars, so at each position you try the alternatives left to right and the first one that matches wins.

tried left to right at every position; the first alternative that matches wins 's |'t |'re |'ve |'m |'ll |'d seven hardcoded English contractions lowercase only, and only with a plain ASCII apostrophe ?\p{L}+ an optional space, then one or more letters from any script this is why the leading space rides along with the word ?\p{N}+ an optional space, then one or more numeric characters letters and numbers can now never merge into one token ?[^\s\p{L}\p{N}]+ an optional space, then anything that is not space, letter or number punctuation gets its own chunk, so dog. can never become one token \s+(?!\S) whitespace up to but NOT including the last whitespace character a negative lookahead, so the final space stays free to join the next word \s+ any whitespace left over the final fallback, which catches trailing spaces and newlines Every chunk is tokenized independently and the results are concatenated, so a merge can never cross a chunk boundary.
Figure 3. The GPT-2 splitting pattern from encoder.py, read the way Karpathy reads it at 57:36. Note it uses the third party regex package, not the standard library re, for the Unicode property classes \p{L} and \p{N}.

Walk it on "Hello world how are you". At the start, Hello is not 's or 't or any of the contractions, but it is "an optional space followed by \p{L} one or more times." What is \p{L}? From the documentation he finds: a letter, any kind of letter from any language. Hello is made of letters, so the alternative matches, and then the match ends, because a whitespace is not a letter. From there a fresh attempt begins, the optional space matches, then the letters, so " world" comes out as the next element. Run findall on the whole thing and you get five elements: Hello, " world", " how", " are", " you".

Why does that matter? "Instead of directly encoding it for tokenization, we are first splitting it up." The text becomes a list of texts, all of these elements are processed independently by the tokenizer, and all of the results of that processing are simply concatenated. Which means, and this is the whole point, "you're only ever finding merges between the elements of this list." Merges can only ever be considered within one element. After all the merging is done per element, the results are joined.

The concrete consequence, in his words: "you are never going to be merging this e with this space, because they are now parts of separate elements of this list." Remember that the single most common pair in his own training text was (101, 32), which is exactly e followed by a space. That merge is now structurally impossible. "Using this regex pattern to chunk up the text is just one way of enforcing that some merges are not to happen."

And the high level intent across the whole pattern: do not merge across letters, across numbers, across punctuation.

\p{N} is "any kind of numeric character in any script," so numbers are separated out from letters. Feed it "Hello world 123 how are you" and world stops matching at the space before 1, because 1 is not a letter, and the numeric group picks it up as its own entity.

What the GPT-2 pattern gets wrong, in his own assessment

The apostrophes are where he stops being neutral. "Why are they doing the apostrophes here? Honestly I think that these are just very common apostrophes that are used. I don't love that they've done this."

Two separate problems. First, the apostrophe is hardcoded as the plain ASCII one. Write house's with a straight quote and the contraction rule splits it out cleanly. Write it with the Unicode typographic apostrophe instead and the rule does not fire, the apostrophe becomes its own thing, and you get different tokenization for what any human reads as the same text.

Second, case. In the GPT-2 docs, where they define the pattern, there is an acknowledgement:

"Should have added re.IGNORECASE so BPE merges can happen for capitalized versions of contractions."

Because they did not, these rules do not separate the apostrophe forms when the letters are uppercase. Lowercase house's splits one way; uppercase HOUSE'S suddenly has the apostrophe come out by itself. "So the tokenization will work differently in uppercase and lowercase, inconsistently separating out these apostrophes. So it feels extremely gnarly and slightly gross, but that's how that works."

And a third, broader worry: "these are quite language specific probably, so I don't know that all the languages for example use or don't use apostrophes, but that would be inconsistently tokenized as a result."

The whitespace alternative with the negative lookahead gets its own careful explanation, because the reason for it is subtle. \s+(?!\S) matches a run of whitespace up to but not including the last whitespace character. Why would you want that? Because the space is always included at the beginning of the word in this scheme, as in " r", " u" and so on. Suppose you have a long run of spaces followed by a word. This alternative catches all the spaces except the last one, so that the last one is free to come around and join with the word, producing " you" rather than stranding it. "The reason that's nice is because " you" is the common token." Add more spaces and you still get " you" at the end, plus a chunk of extra whitespace in front. "Basically the GPT-2 tokenizer really likes to have a space, letters or numbers, and it preens these spaces, and this is just something that it is consistent about."

He finishes with a real world example: feed the pattern a piece of Python code and the resulting list has many elements, because the split fires every time a category changes, "and so there will never be any merges within these elements."

The spaces OpenAI never merged, and the training code nobody released

Here is where he draws a line between what you can reconstruct and what you cannot.

You might assume OpenAI used this pattern to split text into chunks and then ran plain BPE within each chunk. That is not exactly what happened, and you can prove it from the outside. Notice that the runs of spaces in the Python example become whole elements of the list. Those spaces never actually end up merged by OpenAI. Paste the identical chunk into Tiktokenizer with the GPT-2 tokenizer selected and all the spaces are kept independent, all of them token 220. "So I think OpenAI at some point enforced some rule that these spaces would never be merged. There's some additional rules on top of just chunking and BPE that OpenAI is not clear about."

And the reason you have to infer it rather than read it: "The training code for the GPT-2 tokenizer was never released. All we have is the code that I've already shown you, but this code here that they've released is only the inference code for the tokens. This is not the training code. You can't give it a piece of text and train a tokenizer." You can apply their merges to new text. You cannot reproduce how they got those merges. "We don't know exactly how OpenAI trained the tokenizer, but it wasn't as simple as chunk it up and BPE it."

tiktoken, and exactly what GPT-4 changed

tiktoken is OpenAI's official tokenization library. pip install tiktoken, and like encoder.py it is inference only: it tokenizes, it does not train.

Usage is simple enough that he shows it in a few lines, getting GPT-2 tokens out of one encoding and GPT-4 tokens out of the other. And the difference shows up immediately on whitespace. In GPT-2 the runs of spaces remain unmerged. In GPT-4 they merge, which is the same behavior we saw in Tiktokenizer.

The reason is that the regex changed. Go into the installed library and open tiktoken_ext/openai_public.py, which is where the definitions of all the tokenizers OpenAI maintains live. "Necessarily, to do the inference, they had to publish some of the details about the strings."

What you find there for GPT-2 is a pattern with a comment above it saying the original release used the pattern we just read, and that this one "is equivalent, but executes faster." Then scroll down to cl100k_base, the GPT-4 tokenizer, and the pattern has genuinely changed:

# GPT-2 (gpt2, r50k_base), rewritten for speed
r50k_pat_str = r"""'(?:[sdmt]|ll|ve|re)| ?\p{L}++| ?\p{N}++| ?[^\s\p{L}\p{N}]++|\s++$|\s+(?!\S)|\s"""

# GPT-4 (cl100k_base)
pat_str = r"""'(?i:[sdmt]|ll|ve|re)|[^\r\n\p{L}\p{N}]?+\p{L}++|\p{N}{1,3}+| ?[^\s\p{L}\p{N}]++[\r\n]*+|\s++$|\s*[\r\n]|\s+(?!\S)|\s"""

He declines to read the new one character by character. "I'm not going to actually go into the full detail of the pattern change, because honestly this is mind numbing. I would just advise that you pull out ChatGPT and the regex documentation and just step through it." But he names the three changes that matter.

One, case insensitivity. The (?i:...) group around the contractions is a case insensitive match. Which is precisely the fix for the complaint in the GPT-2 docs. "The comment that we saw earlier on, oh we should have used re.IGNORECASE, basically we're now going to be matching these apostrophe s, apostrophe d, apostrophe m, both in lowercase and in uppercase. So that's fixed."

Two, whitespace handling, which he does not detail but which is the change you can see in the Python example: runs of spaces now group into single tokens.

Three, a hard cap on digits. \p{N}{1,3} matches one to three numeric characters and no more. "They will never merge numbers that are more than three digits. Only up to three digits of numbers will ever be merged." Which exists to prevent tokens that are very long number sequences.

And then the honest caveat that he repeats several times in this stretch: "we don't really know why they do any of this stuff, because none of this is documented, and it's just we just get the pattern. So yeah, it is what it is."

Plus the headline number: the vocabulary went from roughly 50,000 to roughly 100,000.

SettingGPT-2 (gpt2, r50k_base)GPT-4 (cl100k_base)
Vocabulary size50,257 (explicit_n_vocab in tiktoken)Roughly 100,000, "roughly double"
Composition of the vocabulary256 raw byte tokens + 50,000 merges + 1 special = 50,257Not stated in the video, same construction
Context length1,024 tokens, up from GPT-1's 512Not covered here
Contraction handlinglowercase only, and the docs admit "should have added re.IGNORECASE"case insensitive via '(?i:[sdmt]|ll|ve|re)
Digit runsUncapped in the pattern, so merges of long digit strings happen by chancecapped at three by \p{N}{1,3}
Whitespace in codeone token per space, all of them token 220, which never mergeruns grouped: four spaces become a single token, seven spaces become a single token
His test string300 tokens185 tokens for the identical text
Special tokens1: <|endoftext|> at id 502565: <|endoftext|> 100257, <|fim_prefix|> 100258, <|fim_middle|> 100259, <|fim_suffix|> 100260, <|endofprompt|> 100276
Training code releasedno, inference only, so the merges cannot be reproducedno, inference only
Rationale documentednono, reverse engineered from the pattern
First merge in trainingNot shownTwo spaces into one token, which becomes token 256
Figure 4. GPT-2 against GPT-4, assembled from what Karpathy reads out of tiktoken_ext/openai_public.py and the GPT-2 paper. The green cells are the three deliberate fixes; the amber ones are the things still undocumented. The whitespace row is most of the reason GPT-4 writes better Python than GPT-2.

Reading encoder.py: their encoder is our vocab, their vocab.bpe is our merges

A brief but genuinely useful walkthrough, because it makes the mapping from the lecture's code to OpenAI's released code explicit.

At the bottom of encoder.py they load two files, encoder.json and vocab.bpe, do some light processing on them, and call the resulting object encoder, which is the tokenizer. Together those two files are the saved tokenizer, and you can download and inspect them yourself.

The translation:

"Using just these two variables you can represent a tokenizer, and you can both do encoding and decoding once you've trained this tokenizer." Our two critical variables and their two files are the same two things.

One thing in their file is a dead end, and he says so rather than pretending it is deep. In addition to the encoder and decoder they also have a byte_encoder and a byte_decoder, "and this is actually unfortunately just kind of a spurious implementation detail and isn't actually deep or interesting in any way." It is a whole separate layer applied serially with the tokenizer: byte encode, then encode, then decode, then byte decode. "It's not that interesting so I won't cover it, and you can step through it if you'd like."

Ignore those two and the file becomes familiar. The meat is their bpe function, and you should recognize its loop: it identifies the bigram, the pair, that should be merged next, then a for loop goes over the sequence merging that pair wherever it finds it, repeating until it runs out of possible merges. Plus an encode and a decode just like ours. "Unfortunately it's a little bit of a messy code that they have, but algorithmically it is identical to what we've built up above."

Special tokens, and the attack surface they create

Beyond tokens that come from raw bytes and BPE merges, you can insert tokens of your own to delimit parts of the data or to give the token stream structure.

Start with the arithmetic of GPT-2's vocabulary, because it resolves cleanly. The length of their encoder object is 50,257. There are 256 raw byte tokens. OpenAI did 50,000 merges. That is 50,256. So what is the last one?

One special token, and you can see its name in the file: <|endoftext|>, id 50256, the very last token in the vocabulary. Its job is to delimit documents in the training set. You tokenize all your documents, you get a stream of tokens in the range 0 to 50,255, and between documents you insert 50256. "We are using this as a signal to the language model that the document has ended, and what follows is going to be unrelated to the document previously."

With the caveat that nothing enforces this. "The language model has to learn this from data. It needs to learn that this token usually means that it should wipe its sort of memory of what came before, and that what came before this token is not actually informative to what comes next. But we are expecting the language model to just learn this."

He demonstrates it in Tiktokenizer. Type the characters of <|endoftext|> one at a time and you get a run of ordinary tokens for the individual pieces, right up until the string is complete, at which point the whole thing collapses into token 50256. And the reason that works is important: "this didn't actually go through the BPE merges. Instead the code that actually outputs the tokens has special case instructions for handling special tokens." Those instructions are not in encoder.py. They are in the tiktoken library, which is implemented in Rust, where you find "all kinds of special case handling for these special tokens that you can register, create, add to the vocabulary, and then it looks for them, and whenever it sees these special tokens like this it will actually come in and swap in that special token. So these things are outside of the typical algorithm of byte pair encoding."

Special tokens get used far more heavily once you leave base language modelling. Refresh Tiktokenizer and the default example uses a fine tuned model's scheme rather than a base model's: pick GPT-3.5-turbo and you see <|im_start|> and <|im_end|> and friends, marking the start and end of every single message in a conversation. "This is short for imaginary monologue start, by the way." Many more tokens are in use there to delimit conversations and keep track of the flow of messages, because you no longer just want to delimit documents, you want to delimit entire conversations between an assistant and a user.

tiktoken documents how to extend it: you can fork cl100k_base, add more special tokens of your own invention with new ids, and the library will correctly swap them in when it sees those strings. Looking at the same definitions file, GPT-2 registers exactly one special token, the end of text. GPT-4 registers five: end of text, three FIM tokens, and one more. FIM is short for fill in the middle, from this paper, which he flags as beyond the scope of the video.

But adding a special token is not free, and this is the part people skip. "When you add special tokens you of course have to do some model surgery to the Transformer and all the parameters involved in that Transformer." Specifically:

"So basically there's some model surgery involved that you have to couple with the tokenization changes, if you are going to add special tokens. But this is a very common operation that people do, especially if they'd like to fine tune the model, for example taking it from a base model to a chat model like ChatGPT."

Your turn: the minbpe exercise

At this point, he says, you have everything you need to build your own GPT-4 tokenizer. He did exactly that while developing the lecture and published the code as minbpe, alongside exercise.md, which breaks the task into four steps that build up to a GPT-4 tokenizer. "Anytime you feel stuck, just reference the minbpe repository. Either the tests could be useful or the repository itself. I try to keep the code fairly clean and understandable."

The acceptance test is behavioral: once you write it, you should be able to reproduce tiktoken's GPT-4 behavior, encoding a string to exactly the right token ids and decoding it back to recover the original. And then one more thing tiktoken cannot do for you: "you should be able to implement your own train function, which tiktoken library does not provide. It's again only inference code. But you could write your own. minbpe does it as well, and that will allow you to train your own token vocabularies."

Then the comparison that makes this section worth the time. Inside minbpe he visualizes two vocabularies side by side. On the left, the GPT-4 merges: the first 256 entries are raw individual bytes, and then the merges GPT-4 actually performed during its training, in order. "The very first merge that GPT-4 did was merge two spaces into a single token, for you know two spaces, and that is token 256."

On the right, the merge order he obtained by training minbpe himself. His training text was the Wikipedia page for Taylor Swift, shipped in the repo as tests/taylorswift.txt, "not because I'm a Swifty, but because that is one of the longest Wikipedia pages apparently that's available. But she is pretty cool."

The two vocabularies line up remarkably well. GPT-4 merged i and n to become in; minbpe did the same thing. His token 259 is " t"; GPT-4 made the same merge a little earlier. "The difference here is, to my understanding, only a difference of the training set. As an example, because I see a lot of whitespace, I suspect that GPT-4 probably had a lot of Python code in its training set, I'm not sure, for the tokenizer, and here we see much less of that of course in the Wikipedia page. So roughly speaking they look the same, and they look the same because they're running the same algorithm."

SentencePiece, and why the open models behave differently

SentencePiece is the other library in wide use, and the reason it matters is that unlike tiktoken it can do both training and inference, efficiently. It supports a number of algorithms for training vocabularies, one of which is the byte pair encoding we have been building. It lives on GitHub under Google, and it is what Llama and the Mistral series use, along with many other models.

The big difference is subtle enough that he insists on working an example, because it is hard to explain in the abstract: the order of operations is reversed.

So BPE runs on code points. And then what happens when you meet a code point you have never seen? That is governed by two settings. Rarity is determined by the character_coverage hyperparameter. Rare code points either get mapped to a special unknown token, <unk>, or, if the byte_fallback option is turned on, they get encoded with UTF-8 and the individual bytes of that encoding are translated into tokens, using special byte tokens that get added to the vocabulary for exactly this purpose.

"So it uses BPE on the code points and then it falls back to bytes for rare code points. Personally I find the tiktoken way significantly cleaner, but it's a subtle but pretty major difference between the way they approach tokenization."

The configuration surface, and the baggage in it

He imports SentencePiece, takes the description of SentencePiece itself as a toy dataset, and writes it to toy.txt, because the library "really likes to have a file."

And then the honest reaction: "what's kind of a little bit crazy about SentencePiece is that there's a ton of options and configurations." The reason is lineage. SentencePiece has been around a while and tries to handle a large diversity of things, "and because it's been around I think it has quite a bit of accumulated historical baggage as well." The training options page is long, and the raw protobuf for the trainer spec is actually the more useful documentation.

Many of the options are simply irrelevant to BPE. His example: --shrinking_factor "is not used in the byte pair encoding algorithm, so this is just an argument that is irrelevant to us. It applies to a different training algorithm."

What he does instead is reverse engineer a known good configuration. "I tried to set up SentencePiece in a way that is very, very similar, as far as I can tell maybe identical hopefully, to the way that Llama 2 was trained." The method: take the tokenizer.model file Meta released, open it with the generated protobuf, inspect all the options, and copy over everything that looked relevant.

The settings that result, and what each category is for:

What the trained vocabulary actually looks like

He trains, loads tok400.model, and inspects the vocabulary, and the ordering is itself informative. SentencePiece lays out its vocabulary in four bands, in this order:

  1. Special tokens. <unk> with id 0, then beginning of sequence as 1 and end of sequence as 2. He set pad_id to negative one, so there is no pad token in this run.
  2. The 256 byte tokens, present because byte_fallback was true in Llama, with their ids.
  3. The merge tokens. "These are the parent nodes in the merges, so we're not seeing the children, we're just seeing the parents and their id."
  4. The individual code point tokens, which come last. These are every code point that occurred in the training set, minus the ones that were extremely rare as determined by character coverage. "If a code point occurred only a single time out of like a million sentences or something like that, then it would be ignored and it would not be added to our vocabulary."

Then he encodes a test string that mixes English and Korean, and gets back both ids and the decoded pieces. Two things jump out.

The Korean characters were not part of the training set, so SentencePiece is meeting code points it has never seen, and they have no token. They would be <unk> tokens. But because byte_fallback is true, it instead encodes them with UTF-8 and uses the byte tokens to represent those bytes. You can see the UTF-8 encoding in the id stream, shifted by three, because of the three special tokens occupying ids 0, 1 and 2 ahead of the byte band.

To prove what the fallback is doing, he turns it off and retrains. All the byte tokens disappear from the vocabulary, and the merge band grows, "because we're not taking up space in the vocab size with all the bytes." Then he encodes the same string and the entire Korean portion collapses to a single 0, because unknown is token zero. "You have to keep in mind that this would feed into your language model. So what is a language model supposed to do when all kinds of different things that are unrecognized, because they're rare, just end up mapping into unk? It's not exactly the property that you want. So that's why I think Llama correctly used byte fallback true, because we definitely want to feed these unknown or rare code points into the model in some manner."

The dummy prefix, and the bold underscore

Two more quirks in the decoded output.

Spaces come back as a bold underscore character rather than a space. "I'm not 100% sure, by the way, why SentencePiece switches whitespace into these bold underscore characters. Maybe it's for visualization, I'm not 100% sure why that happens."

And then: why is there an extra space in front of hello? That comes from add_dummy_prefix, which is true. The documentation says it adds whitespace at the beginning of the text in order to treat world and hello world the same way.

What it is fighting is real. Go back to Tiktokenizer and world as a token by itself has a different id than " world": 1917 against 14. Two different tokens for the language model, which then has to learn from data that they are nearly the same concept. "So to the language model in the tiktoken world, basically words in the beginning of sentences and words in the middle of sentences actually look completely different, and it has to learn that they are roughly the same." The dummy prefix fights that by prepending a space during preprocessing, so both become " world". Llama 2 uses this option too.

His verdict on SentencePiece

He states it plainly as a four part summary:

  1. "There's a lot of historical baggage in SentencePiece, a lot of concepts that I think are slightly confusing and potentially contain foot guns, like this concept of a sentence and its maximum length."
  2. "Otherwise it is fairly commonly used in the industry, because it is efficient and can do both training and inference."
  3. "It has a few quirks, like for example unk token must exist, and the way the byte fallbacks are done, and so on. I don't find particularly elegant."
  4. "Unfortunately I have to say it's not very well documented, so it took me a lot of time working with this myself, just visualizing things and trying to really understand what is happening here."

And the practical instruction that follows from all of that: if you want your tokenization to look identical to Meta's Llama 2, copy paste these settings, exactly as he did. Do not reason it out from the documentation.

How to set the vocabulary size

Now the architectural question he deferred at the thirteen minute mark, answered properly. For this he goes back to gpt.py, the Transformer built in the previous video, and asks where vocab_size actually appears. In that file it was 65, an extremely small number, and it will grow a lot. But it barely appears in most of the layers. There are exactly two places:

The token embedding table. A two dimensional array where vocab_size is the number of rows and each row has n_embd entries, the number of channels in the Transformer. Each vocabulary element gets a vector trained by back propagation. As vocab_size grows, this table grows by rows.

The lm_head linear layer at the very end of the Transformer, which produces the logits that become the probabilities for the next token. "Intuitively we're trying to produce a probability for every single token that might come next at every point in time, and so if we have more and more tokens we need to produce more and more probabilities. So every single token is going to introduce an additional dot product that we have to do in this linear layer."

So why can vocab_size not be infinite? He gives three reasons, and the third is the one people miss.

One, compute. The embedding table grows and the lm_head layer grows, so you do a lot more computation at the output and the final layer becomes more expensive.

Two, undertrained parameters. "If you have a very large vocabulary size, say we have a million tokens, then every one of these tokens is going to come up more and more rarely in the training data, because there's a lot more other tokens all over the place. And so we're going to be seeing fewer and fewer examples for each individual token, and you might be worried that basically the vectors associated with every token will be undertrained as a result, because they just don't come up too often and they don't participate in the forward backward pass."

Three, over squishing. As vocabulary grows, sequences shrink, which is good because you attend to more text. "But also you might be worrying that too large of chunks are being squished into single tokens, and so the model just doesn't have as much time to think per some number of characters in the text. Basically we're squishing too much information into a single token, and then the forward pass of the Transformer is not enough to actually process that information appropriately."

His bottom line: "this is mostly an empirical hyperparameter, and it seems like in state of the art architectures today this is usually in the high 10,000s or somewhere around 100,000 today."

Extending the vocabulary of a model that is already trained

Common operation, mild surgery, and he lays out the recipe.

Why you would do it: when you fine tune for a chat model "a lot more new special tokens get introduced on top of the base model to maintain the metadata and all the structure of conversation objects between a user and an assistant, so that takes a lot of special tokens. You might also try to throw in more special tokens for example for using the browser or any other tool. And so it's very tempting to add a lot of tokens for all kinds of special functionality."

How you do it, which is the same surgery as before stated as a procedure:

  1. Resize the embedding, adding rows for the new tokens.
  2. Initialize those new parameters from scratch, as small random numbers.
  3. Extend the weight inside the final linear layer, so dot products get computed for the new tokens and probabilities can be produced for them.

"So both of these are just a resizing operation. It's a very mild model surgery and can be done fairly easily."

And the training pattern that usually goes with it: "it's quite common that basically you would freeze the base model, you introduce these new parameters, and then you only train these new parameters to introduce new tokens into the architecture. You can freeze arbitrary parts of it, or you can train arbitrary parts of it, and that's totally up to you."

Gist tokens: the design space beyond special tokens

He then opens a door and immediately closes it, flagging that it could be an entire video by itself. "There's an entire design space of applications in terms of introducing new tokens into a vocabulary that go way beyond just adding special tokens and special new functionality."

The example is Learning to Compress Prompts with Gist Tokens. The setup: you are using a language model in a setting that requires very long prompts, and those long prompts slow everything down, because you have to encode them, use them, attend over them. Heavy.

What the paper does instead: introduce a few new tokens, put them in a sequence, and train the model by distillation. Keep the entire model frozen and train only the representations of the new tokens, their embeddings, optimizing so that the behavior of the language model is identical to the model that has the very long prompt. "So it's a compression technique of compressing that very long prompt into those few new gist tokens, and so you can train this, and then at test time you can discard your old prompt and just swap in those tokens, and they sort of stand in for that very long prompt and have an almost identical performance."

Where this sits in the taxonomy: "this is one technique and a class of parameter efficient fine tuning techniques where most of the model is basically fixed, and there's no training of the model weights, there's no training of LoRA or anything like that of new parameters. The parameters that you're training are now just the token embeddings."

Tokenizing images, video and audio

The last structural point before the payoff section, and it is a reassuring one: the Transformer does not need to change.

"There's a lot of momentum in how you actually could construct Transformers that can simultaneously process not just text as the input modality but a lot of other modalities, be it images, videos, audio." The question people ask is whether you have to change the architecture fundamentally. "What a lot of people are starting to converge towards is that you're not changing the architecture. You stick with the Transformer, you just kind of tokenize your input domains and then call it a day and pretend it's just text tokens, and just do everything else in an identical manner."

For images he points at an early paper with a clean graphic showing how you chunk an image into integers, which become the tokens of images. The tokens can be hard tokens, where you force them to be integers, or soft tokens, where you do not require them to be discrete but you do force the representations through a bottleneck, as in autoencoders. The underlying technique the chapter title names is vector quantization, which is how a continuous representation gets turned into a finite codebook of discrete symbols in the first place.

For video he points at OpenAI's Sora report, which "really blew the mind of many people and inspired a lot of people in terms of what's possible." Its framing is exactly parallel: whereas LLMs have text tokens, Sora has visual patches. They came up with a way to chunk videos into tokens with their own vocabularies, "and then you can either process discrete tokens, say with autoregressive models, or even soft tokens with diffusion models." Beyond the scope of this video, but the shape of the idea is the same shape we spent two hours on.

The payoff: every item on the crime sheet, explained

"Now that we have come quite deep into the tokenization algorithm and we understand a lot more about how it works, let's loop back around to the beginning of this video and go through some of these bullet points and really see why they happen."

This is the twenty minutes the rest of the lecture exists to make possible. Each item is a live demo, not an assertion.

Why it cannot spell

Characters are chunked into tokens, and some of those tokens are fairly long. To find the worst case he went into the GPT-4 vocabulary and looked for one of the longer tokens: .DefaultCellStyle turns out to be a single individual token. "That's a lot of characters for a single token, so my suspicion is that there's just too much crammed into this single token, and my suspicion was that the model should not be very good at tasks related to spelling of this single token."

So he asks GPT-4: how many letters L are there in the word .DefaultCellStyle? The prompt is "intentionally done that way," because .DefaultCellStyle is one token, so what you see as sixteen characters the model sees as one opaque symbol.

It gets it wrong. "It doesn't actually know how many Ls are in there. It thinks there are three, and actually there are four, if I'm not getting this wrong myself." Four: one in Default, two in Cell, one in Style.

Why it cannot reverse a string, and the trick that makes it work

Same token, a different character level task. He asks GPT-4 to reverse the string .DefaultCellStyle. It reaches for the code interpreter, he stops it and tells it to just try. "It gave me jumble. It doesn't actually really know how to reverse this string going from right to left, so it gave a wrong result."

Then the fix, and the fix is the proof of the diagnosis. Working with the hypothesis that this is tokenization, he asks for the same reversal in two explicit steps. Step one: print out every single character separated by spaces. Step two: reverse that list.

It tries to use a tool again, he stops it again, and this time it works. "It first produced all the characters, and that was actually correct, and then it reversed them, and that was correct, once it had this."

Why: "somehow it can't reverse it directly, but when you go just first listing it out in order, it can do that somehow, and then once it's broken up this way, this becomes all these individual characters, and so now this is much easier for it to see these individual tokens and reverse them and print them out."

The useful generalization: any character level task gets dramatically easier if you first force the string into one token per character.

Why it is worse in non English languages

Two causes stacked, not one, and he is careful to separate them. "It's not only that the language model sees less non English data during training of the model parameters, but also the tokenizer is not sufficiently trained on non English data."

The measurement: "hello how are you" is 5 tokens. Its translation is 15 tokens. A three times blow up for identical meaning.

And the example that surprises him: 안녕하세요 "is just hello basically in Korean, and that ends up being three tokens. I'm actually kind of surprised by that, because that is a very common phrase there, just the typical greeting of like hello, and that ends up being three tokens, whereas our hello is a single token. So basically everything is a lot more bloated and diffuse, and this is I think partly the reason that the model works worse on other languages."

Why arithmetic is hard

The mechanism is a mismatch between what addition needs and what tokenization provides. "Addition is very sort of like, there's an algorithm that is character level for doing addition. So for example here we would first add the ones and then the tens and then the hundreds. You have to refer to specific parts of these digits." Place value arithmetic requires access to individual digit positions.

"But these numbers are represented completely arbitrarily based on whatever happened to merge or not merge during the tokenization process."

He points at a blog post he rates, Integer tokenization is insane, which systematically explores how numbers tokenize in what he believes is GPT-2. The finding for four digit numbers: a given number may arrive as a single token, or as two tokens split 1 and 3, or 2 and 2, or 3 and 1. All the different numbers get all the different combinations, "and you can imagine this is all completely arbitrary. And the model unfortunately sometimes sees a token for all four digits, sometimes for three, sometimes for two, sometimes for one, and it's in an arbitrary manner. And so this is definitely a headwind, if you will, for the language model, and it's kind of incredible that it can kind of do it and deal with it, but it's also kind of not ideal."

Which is why, when Meta trained Llama 2 with SentencePiece, "they make sure to split up all the digits as an example, and this is partly to improve simple arithmetic kind of performance." Forcing every digit into its own token removes the arbitrariness entirely.

Why GPT-2 was bad at Python

Partly a modelling issue, in the architecture, the dataset and the strength of the model. But also, concretely, tokenization: "as we saw here with the simple Python example, the encoding efficiency of the tokenizer for handling spaces in Python is terrible, and every single space is an individual token, and this dramatically reduces the context length that the model can attend across."

His own verdict on it: "that's almost like a tokenization bug for GPT-2, and that was later fixed with GPT-4."

Why the model halts when it sees <|endoftext|>

"Here's a very strange behavior." He tells GPT-4 to print the string <|endoftext|>, and it says: could you please specify the string. He tells it again that <|endoftext|> is the string. He gives it the string, and then it just does not print it.

"So obviously something is breaking here with respect to the handling of the special token, and I don't actually know what OpenAI is doing under the hood here, and whether they are potentially parsing this as an actual token instead of this just being end of text as individual pieces of it, without the special token handling logic."

His hypothesis is about the allowed_special argument. "It might be that someone, when they're calling encode, they are passing in the allowed_special and they are allowing end of text as a special character in the user prompt. But the user prompt of course is attacker controlled text, so you would hope that they don't really parse or use special tokens from that kind of input."

And then the security point, stated as a general principle: "so your knowledge of these special tokens ends up being an attack surface, potentially. And so if you'd like to confuse LLMs, then just try to give them some special tokens and see if you're breaking something by chance."

Why a trailing space ruins your completion

This one he calls "a really fun one," and it is the clearest demonstration of what it means to be out of distribution.

He goes to the OpenAI playground and picks GPT-3.5-turbo-instruct, deliberately, because that is a completion model rather than a chat model, much closer to a base model: you give it a token sequence and it continues it.

Prompt: Here's a tagline for an ice cream shop. Submit, and you get a reasonable continuation. No problem.

Now the same prompt with a single trailing space before submitting. The playground throws a warning:

"Your text ends in a trailing space, which causes worse performance due to how the API splits text into tokens."

Here is what is happening, worked through against the training data. Imagine you found that completion somewhere on the internet, and the LLM trained on it. The continuation starts with a word, say one beginning with o. In GPT's scheme the space character is always a prefix to these tokens, so what appears in the training data is not an o token. It is a " o" token, where the space is part of the o, and together they are token 8840.

So when you leave the prompt ending at shop and let the model complete, it can sample " o" normally. But when you add your own space, that space is encoded as token 220, a space by itself. "And this token otherwise would be part of the tagline, because if there actually is a tagline here, so " o" is the token. And so this is suddenly out of distribution for the model, because this space is part of the next token, but we're putting it here like this, and the model has seen very, very little data of actual space by itself. And we're asking it to complete the sequence, like add in more tokens, but the problem is that we've sort of begun the first token, and now it's been split up, and now we're out of this distribution, and now arbitrary bad things happen."

The general statement of the lesson: "the LLM is on top of these tokens, and these tokens are text chunks, they're not characters in a way you and I would think of them. They are the atoms of what the LLM is seeing, and there's a bunch of weird stuff that comes out of it."

Partial tokens, and the undocumented machinery for them

He pushes the same idea harder. Take .DefaultCellStyle, the long single token, and truncate it: .DefaultCellSty, without the le.

"I bet you that the model has never in its training set seen .DefaultCellSty without le in there. It's always seen this as a single group, because this is some kind of a function, I'm guessing, I don't actually know what this is part of, this is some kind of API. But I bet you that it's never seen this combination of tokens in its training data, because I think it would be extremely rare."

He pastes it into the playground and asks for a completion. Immediate error:

"The model predicted a completion that begins with a stop sequence, resulting in no output. Consider adjusting your prompt or stop sequences."

"What happened here when I clicked submit is that immediately the model emitted an end of text token, I think, or something like that. It basically predicted the stop sequence immediately, and so it had no completion. And so this is why I'm getting a warning again, because we're off the data distribution and the model is just predicting totally arbitrary things. It's just really confused. Basically this is giving it brain damage. It's never seen this before, it's shocked, and it's predicting end of text or something."

He tries again. This time it completes, but the request gets flagged: "this request may violate our usage policies." "Basically something just goes wrong, and there's something like jank. You can just feel the jank, because the model is extremely unhappy with just this, and it doesn't know how to complete it, because it's never occurred in a training set. In a training set it always appears like this and becomes a single token."

He gives the whole family a name: partial tokens. "Either you sort of complete the first character of the next token, or you have long tokens that you then have just some of the characters off. All of these are kind of like issues with partial tokens."

And then the pointer to the hidden machinery that exists because of this. "If you actually dig into the tiktoken repository, go to the Rust code and search for unstable, and you'll see encode_unstable_native, unstable_tokens, and a lot of special case handling. None of this stuff about unstable tokens is documented anywhere, but there's a ton of code dealing with unstable tokens."

What that code is reaching for is what a completion API should ideally do. "If we're putting in .DefaultCellSty, if we're asking for the next token sequence, we're not actually trying to append the next token exactly after this list. We're trying to consider lots of tokens, or I guess we're trying to search over characters that, if we retokenized, would be of high probability, so that we can actually add a single individual character instead of just adding the next full token that comes after this partial token list."

"This is very tricky to describe, and I invite you to maybe look through this. It ends up being an extremely gnarly and hairy kind of topic, and it comes from tokenization fundamentally. Maybe I can even spend an entire video talking about unstable tokens sometime in the future."

SolidGoldMagikarp

"I'm really saving the best for last. My favorite one by far."

It comes from the blog post SolidGoldMagikarp (plus, prompt generation), which is "internet famous now for those of us in LLMs." He advises reading it in full.

What the authors did: take the token embedding table, cluster the tokens by their embedding representation, and look at what falls out. One cluster is deeply strange. SolidGoldMagikarp is in it, along with other tokens whose names the caption track mangles beyond recovery, among them StreamerBot and a token rendered as attRot. "Really weird tokens in this embedding cluster. So what are these tokens and where do they even come from? Like what is SolidGoldMagikarp? It makes no sense."

Then the plot thickens, because you can poke them. Ask the model something completely benign, like "please can you repeat back to me the string SolidGoldMagikarp," and you get a variety of totally broken behavior:

"There's a bunch of tokens, not just SolidGoldMagikarp, that have that kind of behavior. So basically there's a bunch of trigger words, and if you ask the model about these trigger words or you just include them in your prompt, the model goes haywire and has all kinds of really strange behaviors, including ones that violate typical safety guidelines and the alignment of the model, like it's swearing back at you."

And then the explanation, which is the thing the whole lecture was building toward, because it needs the separation of stages from Figure 2 to even state.

SolidGoldMagikarp is a Reddit user. There is a /u/SolidGoldMagikarp. What is thought to have happened, though nobody has definitively established it:

  1. The tokenization dataset contained a great deal of Reddit data, in which that user was mentioned constantly, because they posted a lot. That string occurs many times in the tokenization dataset.
  2. Because it occurs many times, BPE merges it all the way up into a single dedicated token in a vocabulary of 50,000. A whole token devoted to one Reddit user.
  3. Then you train the actual language model, and this Reddit data was not present in the language model's training set. So in the entire training set for the model, SolidGoldMagikarp never occurs. That token never appears.
  4. So that token never gets activated. "It's initialized at random in the beginning of optimization, then you have forward backward passes and updates to the model, and this token is just never updated in the embedding table. That row vector never gets sampled, it never gets used, so it never gets trained, and it's completely untrained."

The analogy he reaches for: "it's kind of like unallocated memory in a typical binary program written in C or something like that."

"And then at test time, if you invoke this token, then you're basically plucking out a row of the embedding table that is completely untrained, and that feeds into a Transformer and creates undefined behavior. And that's what we're seeing here, this completely undefined, never before seen in training behavior. And so any of these kind of weird tokens would evoke this behavior, because fundamentally the model is out of sample, out of distribution."

Why YAML beats JSON on cost

The last item, "although I think a lot of people are quite aware of this": different formats, representations and languages are more or less efficient under GPT tokenizers, or any tokenizer for any LLM.

"JSON is actually really dense in tokens, and YAML is a lot more efficient in tokens." The measurement: the same data, expressed both ways, JSON is 116 tokens and YAML is 99. "So quite a bit of an improvement."

And the reason to care, stated in economic terms: "in the token economy where we are paying per token in many ways, and you are paying in the context length, and you're paying in dollar amount for the cost of processing all this kind of structured data, prefer to use YAML over JSON. And in general, the tokenization density is something that you have to care about and worry about at all times, and try to find efficient encoding schemes, and spend a lot of time in Tiktokenizer and measure the different token efficiencies of different formats and settings."

GPT-4, cl100k_base 185 tokens 1.00x GPT-2, same string 300 tokens 1.62x hello how are you, English 5 tokens 1.00x the same phrase, translated 15 3.00x YAML 99 tokens 1.00x JSON, the same data 116 tokens 1.17x 0 1x 2x 3x tokens consumed by identical content, relative to the cheaper encoding
Figure 5. The three cost measurements Karpathy reads off the screen, put on one axis. Blue is the cheaper encoding of each pair and amber the dearer one. Every bar is paid for three times over: in context window, in latency, and in dollars per token.
SymptomWhat the tokenizer actually didHis example on cameraWhat to do about it
Cannot count letters in a wordLong strings collapse into one opaque token, so the characters are not visible to the model at all.DefaultCellStyle is a single GPT-4 token; asked how many Ls it holds, GPT-4 says three, the answer is fourForce one token per character first, or hand the job to code
Cannot reverse a stringSame cause. Reversal is character work and the model only has chunksReversing .DefaultCellStyle directly returns jumbleworks in two steps: print every character space separated, then reverse that list
Worse in non English languagesTwo causes stacked: less non English data for the model AND fewer merges learned for that script, so chunks are smaller"hello how are you" is 5 tokens, the translation is 15; 안녕하세요 alone is 3 tokens against 1 for hellobudget 3x the tokens, or pick a model whose tokenizer saw your language
Bad at simple arithmeticDigit groupings are set by corpus frequency, not place value, so the same four digit number may split 1 and 3, or 2 and 2, or 3 and 1127 is one token but 677 is " 6" plus "77"; 804 is " 8" plus "04"; 6773 is " 6" plus "773"Use a calculator or code interpreter. Llama 2 splits every digit into its own token on purpose
GPT-2 wrote poor PythonEvery indentation space is its own token, all of them token 220, and OpenAI blocked them from ever mergingA FizzBuzz snippet shows 220, 220, 220, 220 before " if"fixed in GPT-4 by grouping whitespace runs: four spaces became one token
Model halts on <|endoftext|>Special tokens bypass BPE entirely and are swapped in by string matching, so user text containing one can be parsed as the real tokenAsking GPT-4 to print the literal string <|endoftext|> produces evasion, then nothingtreat it as an attack surface: never allow special tokens from attacker controlled input
Trailing whitespace warningThe space belongs to the front of the NEXT token, so a lone space is token 220 and the sequence goes out of distributionGPT-3.5-turbo-instruct warns on a prompt ending in a space; " o" is token 8840, not an o tokenDo not end a completion prompt with whitespace
Nonsense after a truncated tokenPartial tokens: a prefix of a long token is a sequence the model has essentially never seen.DefaultCellSty makes the playground return "the model predicted a completion that begins with a stop sequence, resulting in no output," then a policy flag on the retryDo not split a token mid chunk. tiktoken's undocumented unstable code paths exist for this
SolidGoldMagikarpThe token exists because the TOKENIZER corpus had it, but the MODEL corpus did not, so its embedding row was never trained and still sits at random initializationAsking GPT-2 to repeat the string produces evasion, hallucination, insults; StreamerBot makes it call you namestrain the tokenizer and the model on the same data. He calls an untrained row "unallocated memory"
JSON costs more than YAMLFormats have different token densities under the same tokenizerThe same data is 116 tokens as JSON and 99 as YAMLMeasure your formats in Tiktokenizer and pick the dense one
Figure 6. The crime sheet from the first five minutes, with the mechanism filled in from the two hours of building. Every example in the third column is one he runs live between 1:51:41 and 2:10:20.

Final recommendations

He closes with a short, blunt summary and a list of what to actually do.

On the stage itself: "I know it's annoying, I know it's irritating, I personally really dislike this stage. What I do have to say at this point is don't brush it off. There's a lot of foot guns, sharp edges here, security issues, AI safety issues, as we saw, plugging in unallocated memory into language models. So it's worth understanding this stage."

And then the line the whole lecture earns: "Eternal glory goes to anyone who can get rid of it." He showed one paper that tried, and hopes a lot more follow.

The recommendations, in order:

  1. If you can, reuse the GPT-4 tokens and vocabulary in your application. That is something you should consider.
  2. Use tiktoken for inference. "It is a very efficient and nice library for inference for BPE." He also likes the byte level BPE that tiktoken and OpenAI use, on the merits.
  3. If you must train your own vocabulary from scratch, then use BPE with SentencePiece. With a long list of caveats: "as I mentioned I'm not a huge fan of SentencePiece. I don't like its byte fallback and I don't like that it's doing BPE on Unicode code points. It also has like a million settings, and I think there's a lot of foot guns here, and I think it's really easy to miscalibrate them and you end up cropping your sentences or something like that because of some type of parameter that you don't fully understand."
  4. Be very careful with the settings. "Try to copy paste exactly what Meta did, or basically spend a lot of time looking at all the hyperparameters and go through the code of SentencePiece and make sure that you have this correct."
  5. And even then he considers the algorithm inferior to byte level BPE. "Maybe the best, if you really need to train your vocabulary, maybe the best thing is to just wait for minbpe to become as efficient as possible, and that's something that maybe I hope to work on."

The thing that does not exist yet, stated as a wish: "really what we want is we want tiktoken but training code, and that is the ideal thing that currently does not exist. And minbpe is an implementation of it, but currently it's in Python."

"There might be an advanced video that has even drier and even more detailed in the future, but for now I think we're going to leave things off here."

Key takeaways

Chapters

Karpathy's own 24 chapters, verbatim. Click any timestamp to jump the player.

Notable quotes

Tokenization is my least favorite part of working with large language models, but unfortunately it is necessary to understand in some detail, because it is fairly hairy, gnarly, and there's a lot of hidden foot guns to be aware of. Andrej Karpathy, the opening sentence, 0:00:00

A lot of the issues that may look like just issues with the neural network architecture or the large language model itself are actually issues with the tokenization, and fundamentally trace back to it. Andrej Karpathy, the thesis of the whole lecture, 0:04:08

For the same concept egg, depending on if it's in the beginning of a sentence, at the end of a sentence, lowercase, uppercase or mixed, all this will be basically very different tokens and different IDs, and the language model has to learn from raw data from all the internet text that it's going to be training on that these are actually all the exact same concept. Andrej Karpathy, on the egg demonstration in Tiktokenizer, 0:08:47

GPT-2 is not very good with Python, and it's not anything to do with coding or the language model itself. Andrej Karpathy, pointing at a column of token 220s in a FizzBuzz snippet, 0:11:49

The improvement in the Python coding ability from GPT-2 to GPT-4 is not just a matter of the language model and the architecture and the details of the optimization, but a lot of the improvement here is also coming from the design of the tokenizer and how it groups characters into tokens. Andrej Karpathy, on grouped whitespace in cl100k_base, 0:14:21

Together, these results establish the viability of tokenization free autoregressive sequence modeling at scale. Andrej Karpathy, reading the conclusion of the MEGABYTE paper out loud, 0:23:06

It's not a tree, it's more like a forest. Andrej Karpathy, on the shape the merges dictionary builds, 0:37:01

The tokenizer is a completely separate object from the large language model itself. Andrej Karpathy, stopping to draw the diagram, 0:39:06

So the tokenization will work differently in uppercase and lowercase, inconsistently separating out these apostrophes. So it feels extremely gnarly and slightly gross, but that's how that works. Andrej Karpathy, on the missing re.IGNORECASE in GPT-2, 1:06:52

The training code for the GPT-2 tokenizer was never released. Andrej Karpathy, on why some of this has to be inferred from behavior, 1:11:01

We don't really know why they do any of this stuff, because none of this is documented, and it's just we just get the pattern. So yeah, it is what it is. Andrej Karpathy, on the GPT-4 regex changes, 1:14:36

In language models we prefer not to do any of it, or at least that is my preference as a deep learning person. You want to not touch your data, you want to keep the raw data as much as possible in a raw form. Andrej Karpathy, on SentencePiece's normalization options, 1:33:09

I would just prefer to treat a file as a giant stream of bytes. Andrej Karpathy, on SentencePiece's concept of a sentence, 1:34:10

It doesn't actually know how many Ls are in there. It thinks there are three, and actually there are four, if I'm not getting this wrong myself. Andrej Karpathy, after asking GPT-4 to count the letters in .DefaultCellStyle, 1:52:41

Your knowledge of these special tokens ends up being an attack surface, potentially. And so if you'd like to confuse LLMs, then just try to give them some special tokens and see if you're breaking something by chance. Andrej Karpathy, on the end of text halting behavior, 1:58:20

Basically this is giving it brain damage. It's never seen this before, it's shocked, and it's predicting end of text or something. Andrej Karpathy, on feeding the playground a partial token, 2:02:26

It's kind of like unallocated memory in a typical binary program written in C. Andrej Karpathy, on an embedding row that was never trained, 2:08:34

Fundamentally the model is out of sample, out of distribution. Andrej Karpathy, the one line explanation of SolidGoldMagikarp, 2:09:06

Eternal glory goes to anyone who can get rid of it. Andrej Karpathy, closing on tokenization itself, 2:10:38

Really what we want is we want tiktoken but training code, and that is the ideal thing that currently does not exist. Andrej Karpathy, the last technical sentence of the lecture, 2:12:13

A note on the caption track

The automatic captions for this video mangle technical names constantly, so if you read the raw transcript below alongside this page, the names you want are: byte pair encoding (captioned variously as "bik pair", "bite pair" and "B paare"), minbpe ("MBP", "M be"), tiktoken ("Tik token", "Tech token"), SentencePiece, Mistral ("mistal"), ASCII ("asky"), bigram ("Byram"), the <unk> token ("an token", "ank"), tok400.model ("talk 400"), <|im_start|> as short for imaginary monologue start ("imaginary mcore start"), .DefaultCellStyle ("default style", "default cell sta"), SolidGoldMagikarp ("sold gold Magikarp"), and YAML ("yl", "theal").

Two numbers in the captions are wrong rather than merely misspelled. The first token of the word "Tokenization" is 30642, not the "3,642" the captions give; Karpathy's own text version of this lecture confirms it. And the captioned description of the twentieth merge, "a merge of 25 and 259 becoming 275," drops a digit from the first id; the structural point it illustrates, that a token minted by an earlier merge can be the parent of a later one, is unaffected.

One thing the captions simply lose: the names in the strange embedding cluster he reads out at 2:05:29. SolidGoldMagikarp and StreamerBot are recoverable, and attRot is probably the third, but the rest are past rescue. The original post has the full list.

What has changed since February 2024

The lecture holds up almost completely, which is itself the point: this is the layer of the stack that moves slowest. Three things are worth knowing if you open openai_public.py today and find more than he found.

The vocabulary doubled again. tiktoken now ships o200k_base alongside cl100k_base, named for roughly 200,000 tokens, with <|endoftext|> at id 199999 and <|endofprompt|> at 200018, plus an o200k_harmony variant that adds more conversation delimiters. Same move as GPT-2 to GPT-4, same trade off he lays out at 1:43:27: shorter sequences bought with a fatter embedding table and a fatter softmax.

The splitting pattern kept evolving in the direction he predicted. o200k_base drops the single ?\p{L}+ letter alternative in favour of two alternatives that treat uppercase led and lowercase led runs separately and fold the contraction suffixes into each with (?i:'s|'t|'re|'ve|'m|'ll|'d)?, which is the case handling complaint from the GPT-2 docs taken a step further. The three digit cap, \p{N}{1,3}, survives untouched.

And the non English token tax is now acknowledged in the source. Above the o200k_base pattern sits a comment that reads, in part: "This regex could be made more efficient. If I was the one working on this encoding, I would have done a few other things differently too, e.g. I think you can allocate tokens more efficiently across languages." That is the 15 tokens against 5 from the payoff section, conceded in a code comment rather than fixed.

What has not changed is the thing he wished for. Byte level BPE is still the standard, nothing tokenization free has displaced it, and minbpe's last commit was July 2024, still in Python. "tiktoken but training code" remains the ideal thing that does not exist, though there is now a community Rust port, gnp/minbpe-rs, linked from his README. The other library you will meet in practice and he does not cover is Hugging Face tokenizers, which is a Rust implementation with Python bindings and the same conceptual shape. And if you want the algorithm in readable Python rather than a Rust core, tiktoken itself ships _educational.py, roughly 220 lines carrying bpe_train, bpe_encode, a SimpleBytePairEncoding class with the same train, encode and decode surface he builds, and a terminal visualiser that colors the tokens. It is the closest official companion to this lecture.

Where this sits in the LLM Learning track

This is Part 2 of the LLM Learning track, the first of the two build it yourself lectures, and it comes first for a structural reason: tokenization is the first thing in the pipeline, and it is the easiest thing to get quietly wrong.

It picks up directly from Part 1. The deep dive into LLMs sketched tokenization in ten minutes and then listed the cognitive quirks that come out of it; this lecture is the long version of that ten minutes, and it retroactively supplies the mechanism for several of the behaviors that page describes. Attention in transformers opened up the machine that consumes these tokens, and the embedding table at the start of that walkthrough is the exact table whose rows get minted, resized and occasionally left untrained here. LLMs in five formulas gave you the memory and efficiency formulas that vocabulary size feeds straight into.

What it sets up is Let's reproduce GPT-2, the other half of this stage. That build assumes a tokenizer exists and loads one, so the two lectures run in pipeline order: tokenizer first, model second. The 50,257 vocabulary size you tune in that video is the number this one explains, including why it is 50,257 rather than 50,256.

Read it before Part 3 as well. John Schulman on RLHF is about teaching a model to hedge instead of hallucinate, and a non trivial share of the hallucinations that look like reasoning failures are the partial token and untrained embedding failures catalogued here. And Chris Olah on mechanistic interpretability works on the embedding and feature structure that this lecture shows you can poison from the outside, simply by training the tokenizer on a corpus the model never sees.

Resources mentioned

The code he builds and the code he reads

Papers

Tools and references

Posts, people and models

Full transcript
[00:00:00] hi everyone so in this video I'd like us to cover the process of tokenization in large language models now you see here that I have a set face and that's because uh tokenization is my least favorite part of working with large language models but unfortunately it is necessary to understand in some detail because it it is fairly hairy gnarly and there's a lot of hidden foot guns to be aware of and a lot of oddness with large language models typically traces back to tokenization so what is tokenization now in my previous video Let's Build GPT from scratch uh we [00:00:31] actually already did tokenization but we did a very naive simple version of tokenization so when you go to the Google colab for that video uh you see here that we loaded our training set and our training set was this uh Shakespeare uh data set now in the beginning the Shakespeare data set is just a large string in Python it's just text and so the question is how do we plug text into large language models and in this case here we created a vocabulary of 65 [00:01:01] possible characters that we saw occur in this string these were the possible characters and we saw that there are 65 of them and then we created a a lookup table for converting from every possible character a little string piece into a token an integer so here for example we tokenized the string High there and we received this sequence of tokens and here we took the first 1,000 characters of our data set and we encoded it into tokens and because it is [00:01:32] this is character level we received 1,000 tokens in a sequence so token 18 47 Etc now later we saw that the way we plug these tokens into the language model is by using an embedding table and so basically if we have 65 possible tokens then this embedding table is going to have 65 rows and roughly speaking we're taking the integer associated with every single sing Le token we're using that as a lookup into this table and we're [00:02:04] plucking out the corresponding row and this row is a uh is trainable parameters that we're going to train using back propagation and this is the vector that then feeds into the Transformer um and that's how the Transformer Ser of perceives every single token so here we had a very naive tokenization process that was a character level tokenizer but in practice in state-ofthe-art uh language models people use a lot more complicated schemes unfortunately uh for constructing these uh token [00:02:34] vocabularies so we're not dealing on the Character level we're dealing on chunk level and the way these um character chunks are constructed is using algorithms such as for example the bik pair in coding algorithm which we're going to go into in detail um and cover in this video I'd like to briefly show you the paper that introduced a bite level encoding as a mechanism for tokenization in the context of large language models and I would say that that's probably the gpt2 paper and if you scroll down here to the section [00:03:05] input representation this is where they cover tokenization the kinds of properties that you'd like the tokenization to have and they conclude here that they're going to have a tokenizer where you have a vocabulary of 50,2 57 possible tokens and the context size is going to be 1,24 tokens so in the in in the attention layer of the Transformer neural network every single token is attending to the previous tokens in the sequence and it's going to see up to 1,24 tokens so tokens [00:03:37] are this like fundamental unit um the atom of uh large language models if you will and everything is in units of tokens everything is about tokens and tokenization is the process for translating strings or text into sequences of tokens and uh vice versa when you go into the Llama 2 paper as well I can show you that when you search token you're going to get get 63 hits um and that's because tokens are again pervasive so here they mentioned that they trained on two trillion tokens of data and so [00:04:08] on so we're going to build our own tokenizer luckily the bite be encoding algorithm is not uh that super complicated and we can build it from scratch ourselves and we'll see exactly how this works before we dive into code I'd like to give you a brief Taste of some of the complexities that come from the tokenization because I just want to make sure that we motivate it sufficiently for why we are doing all this and why this is so gross so tokenization is at the heart of a lot of weirdness in large language models and I would advise that you do not brush it off a lot of the issues that may look [00:04:40] like just issues with the new network architecture or the large language model itself are actually issues with the tokenization and fundamentally Trace uh back to it so if you've noticed any issues with large language models can't you know not able to do spelling tasks very easily that's usually due to tokenization simple string processing can be difficult for the large language model to perform natively uh non-english languages can work much worse and to a large extent this is due to tokenization sometimes llms are bad at [00:05:11] simple arithmetic also can trace be traced to tokenization uh gbt2 specifically would have had quite a bit more issues with python than uh future versions of it due to tokenization there's a lot of other issues maybe you've seen weird warnings about a trailing whites space this is a tokenization issue um if you had asked GPT earlier about solid gold Magikarp and what it is you would see the llm go totally crazy and it would start going off about a completely unrelated tangent topic maybe you've [00:05:41] been told to use yl over Json in structure data all of that has to do with tokenization so basically tokenization is at the heart of many issues I will look back around to these at the end of the video but for now let me just um skip over it a little bit and let's go to this web app um the Tik tokenizer bell.app so I have it loaded here and what I like about this web app is that tokenization is running a sort of live in your browser in JavaScript so you can just type here stuff hello world [00:06:11] and the whole string rokenes so here what we see on uh the left is a string that you put in on the right we're currently using the gpt2 tokenizer we see that this string that I pasted here is currently tokenizing into 300 tokens and here they are sort of uh shown explicitly in different colors for every single token so for example uh this word tokenization became two tokens the token 3,642 and [00:06:44] 1,634 the token um space is is token 318 so be careful on the bottom you can show white space and keep in mind that there are spaces and uh sln new line characters in here but you can hide them for clarity the token space at is token 379 the to the Token space the is 262 Etc so you notice here that the space is part of that uh token [00:07:15] chunk now so this is kind of like how our English sentence broke up and that seems all well and good now now here I put in some arithmetic so we see that uh the token 127 Plus and then token six space 6 followed by 77 so what's happening here is that 127 is feeding in as a single token into the large language model but the um number 677 will actually feed in as two separate tokens and so the large language model [00:07:47] has to sort of um take account of that and process it correctly in its Network and see here 804 will be broken up into two tokens and it's is all completely arbitrary and here I have another example of four-digit numbers and they break up in a way that they break up and it's totally arbitrary sometimes you have um multiple digits single token sometimes you have individual digits as many tokens and it's all kind of pretty arbitrary and coming out of the tokenizer here's another example we have [00:08:17] the string egg and you see here that this became two tokens but for some reason when I say I have an egg you see when it's a space egg it's two token it's sorry it's a single token so just egg by itself in the beginning of a sentence is two tokens but here as a space egg is suddenly a single token uh for the exact same string okay here lowercase egg turns out to be a single token and in particular notice that the color is [00:08:47] different so this is a different token so this is case sensitive and of course a capital egg would also be different tokens and again um this would be two tokens arbitrarily so so for the same concept egg depending on if it's in the beginning of a sentence at the end of a sentence lowercase uppercase or mixed all this will be uh basically very different tokens and different IDs and the language model has to learn from raw data from all the internet text that it's going to be training on that these are actually all the exact same concept [00:09:17] and it has to sort of group them in the parameters of the neural network and understand just based on the data patterns that these are all very similar but maybe not almost exactly similar but but very very similar um after the EG demonstration here I have um an introduction from open a eyes chbt in Korean so manaso Pang uh Etc uh so this is in Korean and the reason I put this here is because you'll notice [00:09:47] that um non-english languages work slightly worse in Chachi part of this is because of course the training data set for Chachi is much larger for English and for everything else but the same is true not just for the large language model itself but also for the tokenizer so when we train the tokenizer we're going to see that there's a training set as well and there's a lot more English than non-english and what ends up happening is that we're going to have a lot more longer tokens for English so how do I put this if you have [00:10:19] a single sentence in English and you tokenize it you might see that it's 10 tokens or something like that but if you translate that sentence into say Korean or Japanese or something else you'll typically see that the number of tokens used is much larger and that's because the chunks here are a lot more broken up so we're using a lot more tokens for the exact same thing and what this does is it bloats up the sequence length of all the documents so you're using up more tokens and then in the attention of the Transformer when these tokens try to [00:10:49] attend each other you are running out of context um in the maximum context length of that Transformer and so basically all the non-english text is stretched out from the perspective of the Transformer and this just has to do with the um trainings that used for the tokenizer and the tokenization itself so it will create a lot bigger tokens and a lot larger groups in English and it will have a lot of little boundaries for all the other non-english text um so if we [00:11:19] translated this into English it would be significantly fewer tokens the final example I have here is a little snippet of python for doing FS buuz and what I'd like you to notice is look all these individual spaces are all separate tokens they are token 220 so uh 220 220 220 220 and then space if is a single token and so what's going on here is that when the Transformer is going to consume or try to uh create [00:11:49] this text it needs to um handle all these spaces individually they all feed in one by one into the entire Transformer in the sequence and so this is being extremely wasteful tokenizing it in this way and so as a result of that gpt2 is not very good with python and it's not anything to do with coding or the language model itself it's just that if he use a lot of indentation using space in Python like we usually do uh you just end up bloating out all the text and it's separated across way too [00:12:19] much of the sequence and we are running out of the context length in the sequence uh that's roughly speaking what's what's happening we're being way too wasteful we're taking up way too much token space now we can also scroll up here and we can change the tokenizer so note here that gpt2 tokenizer creates a token count of 300 for this string here we can change it to CL 100K base which is the GPT for tokenizer and we see that the token count drops to 185 so for the exact same string we are now roughly having the number of tokens and [00:12:49] roughly speaking this is because uh the number of tokens in the GPT 4 tokenizer is roughly double that of the number of tokens in the gpt2 tokenizer so we went went from roughly 50k to roughly 100K now you can imagine that this is a good thing because the same text is now squished into half as many tokens so uh this is a lot denser input to the Transformer and in the Transformer every single token has a finite number of tokens before it that it's going to pay attention to and so what this is doing [00:13:20] is we're roughly able to see twice as much text as a context for what token to predict next uh because of this change but of course just increasing the number of tokens is uh not strictly better infinitely uh because as you increase the number of tokens now your embedding table is um sort of getting a lot larger and also at the output we are trying to predict the next token and there's the soft Max there and that grows as well we're going to go into more detail later on this but there's some kind of a Sweet Spot somewhere where you have a just [00:13:51] right number of tokens in your vocabulary where everything is appropriately dense and still fairly efficient now one thing I would like you to note specifically for the gp4 tokenizer is that the handling of the white space for python has improved a lot you see that here these four spaces are represented as one single token for the three spaces here and then the token SPF and here seven spaces were all grouped into a single token so we're being a lot more efficient in how we represent Python and this was a [00:14:21] deliberate Choice made by open aai when they designed the gp4 tokenizer and they group a lot more space into a single character what this does is this densifies Python and therefore we can attend to more code before it when we're trying to predict the next token in the sequence and so the Improvement in the python coding ability from gbt2 to gp4 is not just a matter of the language model and the architecture and the details of the optimization but a lot of the Improvement here is also coming from [00:14:52] the design of the tokenizer and how it groups characters into tokens okay so let's now start writing some code so remember what we want to do we want to take strings and feed them into language models for that we need to somehow tokenize strings into some integers in some fixed vocabulary and then we will use those integers to make a look up into a lookup table of vectors and feed those vectors into the Transformer as an input now the reason this gets a little bit tricky of course [00:15:22] is that we don't just want to support the simple English alphabet we want to support different kinds of languages so this is anango in Korean which is hello and we also want to support many kinds of special characters that we might find on the internet for example Emoji so how do we feed this text into uh Transformers well how's the what is this text anyway in Python so if you go to the documentation of a string in Python you can see that strings are immutable sequences of Unicode code [00:15:54] points okay what are Unicode code points we can go to PDF so Unicode code points are defined by the Unicode Consortium as part of the Unicode standard and what this is really is that it's just a definition of roughly 150,000 characters right now and roughly speaking what they look like and what integers um represent those characters so it says 150,000 characters across 161 scripts as of right now so if you scroll down here you [00:16:24] can see that the standard is very much alive the latest standard 15.1 in September 2023 and basically this is just a way to define lots of types of characters like for example all these characters across different scripts so the way we can access the unic code code Point given Single Character is by using the or function in Python so for example I can pass in Ord of H and I can see that for the Single Character H the unic [00:16:54] code code point is 104 okay um but this can be arbitr complicated so we can take for example our Emoji here and we can see that the code point for this one is 128,000 or we can take un and this is 50,000 now keep in mind you can't plug in strings here because you uh this doesn't have a single code point it only takes a single uni code code Point character and tells you its integer so in this way we can look [00:17:26] up all the um characters of this specific string and their code points so or of X forx in this string and we get this encoding here now see here we've already turned the raw code points already have integers so why can't we simply just use these integers and not have any tokenization at all why can't we just use this natively as is and just use the code Point well one reason for that of course is that the vocabulary in that case would be quite long so in this [00:17:56] case for Unicode the this is a vocabulary of 150,000 different code points but more worryingly than that I think the Unicode standard is very much alive and it keeps changing and so it's not kind of a stable representation necessarily that we may want to use directly so for those reasons we need something a bit better so to find something better we turn to encodings so if we go to the Wikipedia page here we see that the Unicode consortion defines three types of encodings utf8 UTF 16 and UTF 32 these [00:18:27] encoding are the way by which we can take Unicode text and translate it into binary data or by streams utf8 is by far the most common uh so this is the utf8 page now this Wikipedia page is actually quite long but what's important for our purposes is that utf8 takes every single Cod point and it translates it to a by stream and this by stream is between one to four bytes so it's a variable length encoding so depending on the Unicode Point according to the schema you're [00:18:58] going to end up with between 1 to four bytes for each code point on top of that there's utf8 uh utf16 and UTF 32 UTF 32 is nice because it is fixed length instead of variable length but it has many other downsides as well so the full kind of spectrum of pros and cons of all these different three encodings are beyond the scope of this video I just like to point out that I enjoyed this block post and this block post at the end of it also has a number of references that can be quite useful [00:19:29] uh one of them is uh utf8 everywhere Manifesto um and this Manifesto describes the reason why utf8 is significantly preferred and a lot nicer than the other encodings and why it is used a lot more prominently um on the internet one of the major advantages just just to give you a sense is that utf8 is the only one of these that is backwards compatible to the much simpler asky encoding of text um but I'm not going to go into the full detail in this video so suffice to say that we like the [00:20:01] utf8 encoding and uh let's try to take the string and see what we get if we encoded into utf8 the string class in Python actually has do encode and you can give it the encoding which is say utf8 now we get out of this is not very nice because this is the bytes is a bytes object and it's not very nice in the way that it's printed so I personally like to take it through list because then we actually get the raw B of this uh encoding so this is the raw [00:20:32] byes that represent this string according to the utf8 en coding we can also look at utf16 we get a slightly different by stream and we here we start to see one of the disadvantages of utf16 you see how we have zero Z something Z something Z something we're starting to get a sense that this is a bit of a wasteful encoding and indeed for simple asky characters or English characters here uh we just have the structure of 0 something Z something and it's not exactly nice same for UTF 32 when we [00:21:04] expand this we can start to get a sense of the wastefulness of this encoding for our purposes you see a lot of zeros followed by something and so uh this is not desirable so suffice it to say that we would like to stick with utf8 for our purposes however if we just use utf8 naively these are by streams so that would imply a vocabulary length of only 256 possible tokens uh but this this vocabulary size is very very small what [00:21:35] this is going to do if we just were to use it naively is that all of our text would be stretched out over very very long sequences of bytes and so um what what this does is that certainly the embeding table is going to be tiny and the prediction at the top at the final layer is going to be very tiny but our sequences are very long and remember that we have pretty finite um context length and the attention that we can support in a transformer for computational reasons and so we only [00:22:05] have as much context length but now we have very very long sequences and this is just inefficient and it's not going to allow us to attend to sufficiently long text uh before us for the purposes of the next token prediction task so we don't want to use the raw bytes of the utf8 encoding we want to be able to support larger vocabulary size that we can tune as a hyper but we want to stick with the utf8 encoding of these strings so what do we do well the answer of course is we turn [00:22:35] to the bite pair encoding algorithm which will allow us to compress these bite sequences um to a variable amount so we'll get to that in a bit but I just want to briefly speak to the fact that I would love nothing more than to be able to feed raw bite sequences into uh language models in fact there's a paper about how this could potentially be done uh from Summer last last year now the problem is you actually have to go in and you have to modify the Transformer architecture because as I mentioned you're going to have a problem where the [00:23:06] attention will start to become extremely expensive because the sequences are so long and so in this paper they propose kind of a hierarchical structuring of the Transformer that could allow you to just feed in raw bites and so at the end they say together these results establish the viability of tokenization free autor regressive sequence modeling at scale so tokenization free would indeed be amazing we would just feed B streams directly into our models but unfortunately I don't know that this has really been proven out yet by [00:23:36] sufficiently many groups and a sufficient scale uh but something like this at one point would be amazing and I hope someone comes up with it but for now we have to come back and we can't feed this directly into language models and we have to compress it using the B paare encoding algorithm so let's see how that works so as I mentioned the B paare encoding algorithm is not all that complicated and the Wikipedia page is actually quite instructive as far as the basic idea goes go what we're doing is we have some kind of a input sequence uh like for example here we have only four elements in our vocabulary a b c and d [00:24:06] and we have a sequence of them so instead of bytes let's say we just have four a vocab size of four the sequence is too long and we'd like to compress it so what we do is that we iteratively find the pair of uh tokens that occur the most frequently and then once we've identified that pair we repl replace that pair with just a single new token that we append to our vocabulary so for example here the bite pair AA occurs [00:24:36] most often so we mint a new token let's call it capital Z and we replace every single occurrence of AA by Z so now we have two Z's here so here we took a sequence of 11 characters with vocabulary size four and we've converted it to a um sequence of only nine tokens but now with a vocabulary of five because we have a fifth vocabulary element that we just created and it's Z standing for concatination of AA and we [00:25:07] can again repeat this process so we again look at the sequence and identify the pair of tokens that are most frequent let's say that that is now AB well we are going to replace AB with a new token that we meant call Y so y becomes ab and then every single occurrence of ab is now replaced with y so we end up with this so now we only have 1 2 3 4 5 6 seven characters in our sequence but we have not just um four [00:25:40] vocabulary elements or five but now we have six and for the final round we again look through the sequence find that the phrase zy or the pair zy is most common and replace it one more time with another um character let's say x so X is z y and we replace all curses of zy and we get this following sequence so basically after we have gone through this process instead of having a um sequence of 11 uh tokens with a vocabulary length of [00:26:13] four we now have a sequence of 1 2 3 four five tokens but our vocabulary length now is seven and so in this way we can iteratively compress our sequence I we Mint new tokens so in the in the exact same way we start we start out with bite sequences so we have 256 vocabulary size but we're now going to go through these and find the bite pairs that occur the most and we're going to iteratively start minting new tokens [00:26:44] appending them to our vocabulary and replacing things and in this way we're going to end up with a compressed training data set and also an algorithm for taking any arbitrary sequence and encoding it using this uh vocabul and also decoding it back to Strings so let's now Implement all that so here's what I did I went to this block post that I enjoyed and I took the first paragraph and I copy pasted it here into text so this is one very long line here now to get the tokens as I [00:27:15] mentioned we just take our text and we encode it into utf8 the tokens here at this point will be a raw bites single stream of bytes and just so that it's easier to work with instead of just a bytes object I'm going to convert all those bytes to integers and then create a list of it just so it's easier for us to manipulate and work with in Python and visualize and here I'm printing all of that so this is the original um this is the original paragraph and its length [00:27:45] is 533 uh code points and then here are the bytes encoded in ut utf8 and we see that this has a length of 616 bytes at this point or 616 tokens and the reason this is more is because a lot of these simple asky characters or simple characters they just become a single bite but a lot of these Unicode more complex characters become multiple bytes up to four and so we are expanding that size so now what we'd like to do as a first step of the algorithm is we'd like [00:28:16] to iterate over here and find the pair of bites that occur most frequently because we're then going to merge it so if you are working long on a notebook on a side then I encourage you to basically click on the link find this notebook and try to write that function yourself otherwise I'm going to come here and Implement first the function that finds the most common pair okay so here's what I came up with there are many different ways to implement this but I'm calling the function get stats it expects a list of integers I'm using a dictionary to keep track of basically the counts and [00:28:46] then this is a pythonic way to iterate consecutive elements of this list uh which we covered in the previous video and then here I'm just keeping track of just incrementing by one um for all the pairs so if I call this on all the tokens here then the stats comes out here so this is the dictionary the keys are these topples of consecutive elements and this is the count so just to uh print it in a slightly better way this is one way that I like to do that [00:29:17] where you it's a little bit compound here so you can pause if you like but we iterate all all the items the items called on dictionary returns pairs of key value and instead I create a list here of value key because if it's a value key list then I can call sort on it and by default python will uh use the first element which in this case will be value to sort by if it's given tles and then reverse so it's descending and [00:29:48] print that so basically it looks like 101 comma 32 was the most commonly occurring consecutive pair and it occurred 20 times we can double check that that makes reasonable sense so if I just search 10132 then you see that these are the 20 occurrences of that um pair and if we'd like to take a look at what exactly that pair is we can use Char which is the opposite of or in Python so we give it a um unic code Cod point so 101 and of 32 [00:30:22] and we see that this is e and space so basically there's a lot of E space here meaning that a lot of these words seem to end with e so here's eace as an example so there's a lot of that going on here and this is the most common pair so now that we've identified the most common pair we would like to iterate over this sequence we're going to Mint a new token with the ID of 256 right because these tokens currently go from Z to 255 so when we create a new token it will have an ID of [00:30:52] 256 and we're going to iterate over this entire um list and every every time we see 101 comma 32 we're going to swap that out for 256 so let's Implement that now and feel free to uh do that yourself as well so first I commented uh this just so we don't pollute uh the notebook too much this is a nice way of in Python obtaining the highest ranking pair so we're basically calling the Max on this [00:31:23] dictionary stats and this will return the maximum key and then the question is how does it rank keys so you can provide it with a function that ranks keys and that function is just stats. getet uh stats. getet would basically return the value and so we're ranking by the value and getting the maximum key so it's 101 comma 32 as we saw now to actually merge 10132 um this is the function that I wrote but again there are many different [00:31:53] versions of it so we're going to take a list of IDs and the the pair that we want to replace and that pair will be replaced with the new index idx so iterating through IDs if we find the pair swap it out for idx so we create this new list and then we start at zero and then we go through this entire list sequentially from left to right and here we are checking for equality at the current position with the pair um so here we are checking that the [00:32:23] pair matches now here is a bit of a tricky condition that you have to append if you're trying to be careful and that is that um you don't want this here to be out of Bounds at the very last position when you're on the rightmost element of this list otherwise this would uh give you an autof bounds error so we have to make sure that we're not at the very very last element so uh this would be false for that so if we find a match we append to this new list that replacement index and we increment the [00:32:53] position by two so we skip over that entire pair but otherwise if we we haven't found a matching pair we just sort of copy over the um element at that position and increment by one then return this so here's a very small toy example if we have a list 566 791 and we want to replace the occurrences of 67 with 99 then calling this on that will give us what we're asking for so here the 67 is replaced with 99 so now I'm going to uncomment this [00:33:23] for our actual use case where we want to take our tokens we want to take the top pair here and replace it with 256 to get tokens to if we run this we get the following so recall that previously we had a length 616 in this list and now we have a length 596 right so this decreased by 20 which makes sense because there are 20 occurrences moreover we can try to find 256 here and [00:33:55] we see plenty of occurrences on off it and moreover just double check there should be no occurrence of 10132 so this is the original array plenty of them and in the second array there are no occurrences of 1032 so we've successfully merged this single pair and now we just uh iterate this so we are going to go over the sequence again find the most common pair and replace it so let me now write a y Loop that uses these functions to do this um sort of iteratively and how many times do we do it four well that's totally up to us as [00:34:26] a hyper parameter the more um steps we take the larger will be our vocabulary and the shorter will be our sequence and there is some sweet spot that we usually find works the best in practice and so this is kind of a hyperparameter and we tune it and we find good vocabulary sizes as an example gp4 currently uses roughly 100,000 tokens and um bpark that those are reasonable numbers currently instead the are large language models so let me now write uh putting putting it all together and uh iterating these steps [00:34:58] okay now before we dive into the Y loop I wanted to add one more cell here where I went to the block post and instead of grabbing just the first paragraph or two I took the entire block post and I stretched it out in a single line and basically just using longer text will allow us to have more representative statistics for the bite Pairs and we'll just get a more sensible results out of it because it's longer text um so here we have the raw text we encode it into bytes using the utf8 encoding and then here as before we are just [00:35:30] changing it into a list of integers in Python just so it's easier to work with instead of the raw byes objects and then this is the code that I came up with uh to actually do the merging in Loop these two functions here are identical to what we had above I only included them here just so that you have the point of reference here so uh these two are identical and then this is the new code that I added so the first first thing we want to do is we want to decide on the final vocabulary size that we want our [00:36:01] tokenizer to have and as I mentioned this is a hyper parameter and you set it in some way depending on your best performance so let's say for us we're going to use 276 because that way we're going to be doing exactly 20 merges and uh 20 merges because we already have 256 tokens for the raw bytes and to reach 276 we have to do 20 merges uh to add 20 new tokens here uh this is uh one way in Python to just create a copy of a list [00:36:31] so I'm taking the tokens list and by wrapping it in a list python will construct a new list of all the individual elements so this is just a copy operation then here I'm creating a merges uh dictionary so this merges dictionary is going to maintain basically the child one child two mapping to a new uh token and so what we're going to be building up here is a binary tree of merges but actually it's not exactly a tree because a tree would have a single root node with a bunch of [00:37:01] leaves for us we're starting with the leaves on the bottom which are the individual bites those are the starting 256 tokens and then we're starting to like merge two of them at a time and so it's not a tree it's more like a forest um uh as we merge these elements so for 20 merges we're going to find the most commonly occurring pair we're going to Mint a new token integer for it so I here will start at zero so we'll going to start at 256 we're going to print [00:37:32] that we're merging it and we're going to replace all of the occurrences of that pair with the new new lied token and we're going to record that this pair of integers merged into this new integer so running this gives us the following output so we did 20 merges and for example the first merge was exactly as before the 10132 um tokens merging into a new token 2556 now keep in mind that the [00:38:04] individual uh tokens 101 and 32 can still occur in the sequence after merging it's only when they occur exactly consecutively that that becomes 256 now um and in particular the other thing to notice here is that the token 256 which is the newly minted token is also eligible for merging so here on the bottom the 20th merge was a merge of 25 and 259 becoming 275 so every time we replace these tokens they become eligible for merging in the next round of data ration so [00:38:35] that's why we're building up a small sort of binary Forest instead of a single individual tree one thing we can take a look at as well is we can take a look at the compression ratio that we've achieved so in particular we started off with this tokens list um so we started off with 24,000 bytes and after merging 20 times uh we now have only 19,000 um tokens and so therefore the compression ratio simply just dividing the two is roughly 1.27 so that's the [00:39:06] amount of compression we were able to achieve of this text with only 20 merges um and of course the more vocabulary elements you add uh the greater the compression ratio here would be finally so that's kind of like um the training of the tokenizer if you will now 1 Point I wanted to make is that and maybe this is a diagram that can help um kind of illustrate is that tokenizer is a completely separate object from the large language model itself so [00:39:37] everything in this lecture we're not really touching the llm itself uh we're just training the tokenizer this is a completely separate pre-processing stage usually so the tokenizer will have its own training set just like a large language model has a potentially different training set so the tokenizer has a training set of documents on which you're going to train the tokenizer and then and um we're performing The Bite pair encoding algorithm as we saw above to train the vocabulary of this tokenizer so it has its own training set it is a pre-processing stage that you would run a single time in the beginning [00:40:09] um and the tokenizer is trained using bipar coding algorithm once you have the tokenizer once it's trained and you have the vocabulary and you have the merges uh we can do both encoding and decoding so these two arrows here so the tokenizer is a translation layer between raw text which is as we saw the sequence of Unicode code points it can take raw text and turn it into a token sequence and vice versa it can take a token sequence and translate it back into raw [00:40:40] text so now that we have trained uh tokenizer and we have these merges we are going to turn to how we can do the encoding and the decoding step if you give me text here are the tokens and vice versa if you give me tokens here's the text once we have that we can translate between these two Realms and then the language model is going to be trained as a step two afterwards and typically in a in a sort of a state-of-the-art application you might take all of your training data for the language model and you might run it through the tokenizer and sort of [00:41:10] translate everything into a massive token sequence and then you can throw away the raw text you're just left with the tokens themselves and those are stored on disk and that is what the large language model is actually reading when it's training on them so this one approach that you can take as a single massive pre-processing step a stage um so yeah basically I think the most important thing I want to get across is that this is completely separate stage it usually has its own entire uh training set you may want to have those training sets be different between the tokenizer and the logge language model so for example when [00:41:41] you're training the tokenizer as I mentioned we don't just care about the performance of English text we care about uh multi many different languages and we also care about code or not code so you may want to look into different kinds of mixtures of different kinds of languages and different amounts of code and things like that because the amount of different language that you have in your tokenizer training set will determine how many merges of it there will be and therefore that determines the density with which uh this type of [00:42:11] data is um sort of has in the token space and so roughly speaking intuitively if you add some amount of data like say you have a ton of Japanese data in your uh tokenizer training set then that means that more Japanese tokens will get merged and therefore Japanese will have shorter sequences uh and that's going to be beneficial for the large language model which has a finite context length on which it can work on in in the token space uh so hopefully that makes sense so we're now going to turn to encoding [00:42:41] and decoding now that we have trained a tokenizer so we have our merges and now how do we do encoding and decoding okay so let's begin with decoding which is this Arrow over here so given a token sequence let's go through the tokenizer to get back a python string object so the raw text so this is the function that we' like to implement um we're given the list of integers and we want to return a python string if you'd like uh try to implement this function yourself it's a fun exercise otherwise I'm going to start uh pasting in my own [00:43:11] solution so there are many different ways to do it um here's one way I will create an uh kind of pre-processing variable that I will call vocab and vocab is a mapping or a dictionary in Python for from the token uh ID to the bytes object for that token so we begin with the raw bytes for tokens from 0 to 255 and then we go in order of all the merges and we sort of uh populate this vocab list by doing an [00:43:42] addition here so this is the basically the bytes representation of the first child followed by the second one and remember these are bytes objects so this addition here is an addition of two bytes objects just concatenation so that's what we get here one tricky thing to be careful with by the way is that I'm iterating a dictionary in Python using a DOT items and uh it really matters that this runs in the order in which we inserted items into the merous dictionary luckily [00:44:13] starting with python 3.7 this is guaranteed to be the case but before python 3.7 this iteration may have been out of order with respect to how we inserted elements into merges and this may not have worked but we are using an um modern python so we're okay and then here uh given the IDS the first thing we're going to do is get the tokens so the way I implemented this here is I'm taking I'm iterating over all the IDS I'm using vocap to look up their bytes and then here this is one [00:44:44] way in Python to concatenate all these bytes together to create our tokens and then these tokens here at this point are raw bytes so I have to decode using UTF F now back into python strings so previously we called that encode on a string object to get the bytes and now we're doing it Opposite we're taking the bytes and calling a decode on the bytes object to get a string in Python and then we can return text so um this is how we can do it now [00:45:16] this actually has a um issue um in the way I implemented it and this could actually throw an error so try to think figure out why this code could actually result in an error if we plug in um uh some sequence of IDs that is unlucky so let me demonstrate the issue when I try to decode just something like 97 I am going to get letter A here back so nothing too crazy happening but when I try to decode 128 as a single element [00:45:48] the token 128 is what in string or in Python object uni Cod decoder utfa can't Decode by um 0x8 which is this in HEX in position zero invalid start bite what does that mean well to understand what this means we have to go back to our utf8 page uh that I briefly showed earlier and this is Wikipedia utf8 and basically there's a specific schema that utfa bytes take so in particular if you have a multi-te object for some of the [00:46:19] Unicode characters they have to have this special sort of envelope in how the encoding works and so what's happening here is that invalid start pite that's because 128 the binary representation of it is one followed by all zeros so we have one and then all zero and we see here that that doesn't conform to the format because one followed by all zero just doesn't fit any of these rules so to speak so it's an invalid start bite which is byte one this one must have a [00:46:50] one following it and then a zero following it and then the content of your uni codee in x here so basically we don't um exactly follow the utf8 standard and this cannot be decoded and so the way to fix this um is to use this errors equals in bytes. decode function of python and by default errors is strict so we will throw an error if um it's not valid utf8 bytes encoding [00:47:20] but there are many different things that you could put here on error handling this is the full list of all the errors that you can use and in particular instead of strict let's change it to replace and that will replace uh with this special marker this replacement character so errors equals replace and now we just get that character back so basically not every single by sequence is valid utf8 and if it happens that your large [00:47:51] language model for example predicts your tokens in a bad manner then they might not fall into valid utf8 and then we won't be able to decode them so the standard practice is to basically uh use errors equals replace and this is what you will also find in the openai um code that they released as well but basically whenever you see um this kind of a character in your output in that case uh something went wrong and the LM output not was not valid uh sort of sequence of [00:48:21] tokens okay and now we're going to go the other way so we are going to implement this Arrow right here where we are going to be given a string and we want to encode it into tokens so this is the signature of the function that we're interested in and um this should basically print a list of integers of the tokens so again uh try to maybe implement this yourself if you'd like a fun exercise uh and pause here otherwise I'm going to start putting in my solution so again there are many ways to do this so um this is one of the ways [00:48:53] that sort of I came came up with so the first thing we're going to do is we are going to uh take our text encode it into utf8 to get the raw bytes and then as before we're going to call list on the bytes object to get a list of integers of those bytes so those are the starting tokens those are the raw bytes of our sequence but now of course according to the merges dictionary above and recall this was the merges some of the bytes may be merged [00:49:23] according to this lookup in addition to that remember that the merges was built from top to bottom and this is sort of the order in which we inserted stuff into merges and so we prefer to do all these merges in the beginning before we do these merges later because um for example this merge over here relies on the 256 which got merged here so we have to go in the order from top to bottom sort of if we are going to be merging anything now we expect to be doing a few merges so we're going to be doing W [00:49:54] true um and now we want to find a pair of byes that is consecutive that we are allowed to merge according to this in order to reuse some of the functionality that we've already written I'm going to reuse the function uh get stats so recall that get stats uh will give us the we'll basically count up how many times every single pair occurs in our sequence of tokens and return that as a dictionary and the dictionary was a mapping from all the different uh by [00:50:25] pairs to the number of times that they occur right um at this point we don't actually care how many times they occur in the sequence we only care what the raw pairs are in that sequence and so I'm only going to be using basically the keys of the dictionary I only care about the set of possible merge candidates if that makes sense now we want to identify the pair that we're going to be merging at this stage of the loop so what do we want we want to find the pair or like the a key inside stats that has the lowest index [00:50:57] in the merges uh dictionary because we want to do all the early merges before we work our way to the late merges so again there are many different ways to implement this but I'm going to do something a little bit fancy here so I'm going to be using the Min over an iterator in Python when you call Min on an iterator and stats here as a dictionary we're going to be iterating the keys of this dictionary in Python so we're looking at all the pairs inside [00:51:27] stats um which are all the consecutive Pairs and we're going to be taking the consecutive pair inside tokens that has the minimum what the Min takes a key which gives us the function that is going to return a value over which we're going to do the Min and the one we care about is we're we care about taking merges and basically getting um that pairs index so basically for any pair inside [00:51:57] stats we are going to be looking into merges at what index it has and we want to get the pair with the Min number so as an example if there's a pair 101 and 32 we definitely want to get that pair uh we want to identify it here and return it and pair would become 10132 if it occurs and the reason that I'm putting a float INF here as a fall back is that in the get function when we call uh when we basically consider a pair that doesn't occur in the merges then that pair is [00:52:29] not eligible to be merged right so if in the token sequence there's some pair that is not a merging pair it cannot be merged then uh it doesn't actually occur here and it doesn't have an index and uh it cannot be merged which we will denote as float INF and the reason Infinity is nice here is because for sure we're guaranteed that it's not going to participate in the list of candidates when we do the men so uh so this is one way to do it so B basically long story short this Returns the most eligible merging candidate pair uh that occurs in [00:53:01] the tokens now one thing to be careful with here is this uh function here might fail in the following way if there's nothing to merge then uh uh then there's nothing in merges um that satisfi that is satisfied anymore there's nothing to merge everything just returns float imps and then the pair I think will just become the very first element of stats um but this pair is not actually a mergeable pair it just becomes the first [00:53:31] pair inside stats arbitrarily because all of these pairs evaluate to float in for the merging Criterion so basically it could be that this this doesn't look succeed because there's no more merging pairs so if this pair is not in merges that was returned then this is a signal for us that actually there was nothing to merge no single pair can be merged anymore in that case we will break out um nothing else can be merged you may come up with a different implementation by the way this is kind [00:54:01] of like really trying hard in Python um but really we're just trying to find a pair that can be merged with the lowest index here now if we did find a pair that is inside merges with the lowest index then we can merge it so we're going to look into the merger dictionary for that pair to look up the index and we're going to now merge that into that index so we're going to do tokens equals and we're going to [00:54:32] replace the original tokens we're going to be replacing the pair pair and we're going to be replacing it with index idx and this returns a new list of tokens where every occurrence of pair is replaced with idx so we're doing a merge and we're going to be continuing this until eventually nothing can be merged we'll come out here and we'll break out and here we just return tokens and so that that's the implementation I think so hopefully this runs okay cool um yeah and this looks uh [00:55:02] reasonable so for example 32 is a space in asky so that's here um so this looks like it worked great okay so let's wrap up this section of the video at least I wanted to point out that this is not quite the right implementation just yet because we are leaving out a special case so in particular if uh we try to do this this would give us an error and the issue is that um if we only have a single character or an empty string then stats is empty and that causes an issue inside Min so one way to fight this is [00:55:32] if L of tokens is at least two because if it's less than two it's just a single token or no tokens then let's just uh there's nothing to merge so we just return so that would fix uh that case Okay and then second I have a few test cases here for us as well so first let's make sure uh about or let's note the following if we take a string and we try to encode it and then decode it back you'd expect to get the same string back right is that true for all [00:56:04] strings so I think uh so here it is the case and I think in general this is probably the case um but notice that going backwards is not is not you're not going to have an identity going backwards because as I mentioned us not all token sequences are valid utf8 uh sort of by streams and so so therefore you're some of them can't even be decodable um so this only goes in One Direction but for that one direction we can check uh here if we take the [00:56:34] training text which is the text that we train to tokenizer around we can make sure that when we encode and decode we get the same thing back which is true and here I took some validation data so I went to I think this web page and I grabbed some text so this is text that the tokenizer has not seen and we can make sure that this also works um okay so that gives us some confidence that this was correctly implemented so those are the basics of the bite pair encoding algorithm we saw how we can uh take some training set train a tokenizer the parameters of this tokenizer really [00:57:05] are just this dictionary of merges and that basically creates the little binary Forest on top of raw bites once we have this the merges table we can both encode and decode between raw text and token sequences so that's the the simplest setting of The tokenizer what we're going to do now though is we're going to look at some of the St the art lar language models and the kinds of tokenizers that they use and we're going to see that this picture complexifies very quickly so we're going to go through the details of this comp complexification one at a time so let's [00:57:37] kick things off by looking at the GPD Series so in particular I have the gpt2 paper here um and this paper is from 2019 or so so 5 years ago and let's scroll down to input representation this is where they talk about the tokenizer that they're using for gpd2 now this is all fairly readable so I encourage you to pause and um read this yourself but this is where they motivate the use of the bite pair encoding algorithm on the bite level representation of utf8 [00:58:07] encoding so this is where they motivate it and they talk about the vocabulary sizes and everything now everything here is exactly as we've covered it so far but things start to depart around here so what they mention is that they don't just apply the naive algorithm as we have done it and in particular here's a example suppose that you have common words like dog what will happen is that dog of course occurs very frequently in the text and it occurs right next to all kinds of punctuation as an example so doc dot dog exclamation mark dog [00:58:39] question mark Etc and naively you might imagine that the BP algorithm could merge these to be single tokens and then you end up with lots of tokens that are just like dog with a slightly different punctuation and so it feels like you're clustering things that shouldn't be clustered you're combining kind of semantics with uation and this uh feels suboptimal and indeed they also say that this is suboptimal according to some of the experiments so what they want to do is they want to top down in a manual way enforce that some types of um characters [00:59:09] should never be merged together um so they want to enforce these merging rules on top of the bite PA encoding algorithm so let's take a look um at their code and see how they actually enforce this and what kinds of mergy they actually do perform so I have to to tab open here for gpt2 under open AI on GitHub and when we go to Source there is an encoder thatp now I don't personally love that they call it encoder dopy because this is the tokenizer and the tokenizer can do both [00:59:39] encode and decode uh so it feels kind of awkward to me that it's called encoder but that is the tokenizer and there's a lot going on here and we're going to step through it in detail at one point for now I just want to focus on this part here the create a rigix pattern here that looks very complicated and we're going to go through it in a bit uh but this is the core part that allows them to enforce rules uh for what parts of the text Will Never Be merged for sure now notice that re. compile here is a little bit misleading because we're [01:00:10] not just doing import re which is the python re module we're doing import reex as re and reex is a python package that you can install P install r x and it's basically an extension of re so it's a bit more powerful re um so let's take a look at this pattern and what it's doing and why this is actually doing the separation that they are looking for okay so I've copy pasted the pattern here to our jupit notebook where we left off and let's take this pattern for a spin so in the exact same way that [01:00:42] their code does we're going to call an re. findall for this pattern on any arbitrary string that we are interested so this is the string that we want to encode into tokens um to feed into n llm like gpt2 so what exactly is this doing well re. findall will take this pattern and try to match it against a string um the way this works is that you are going from left to right in the string and you're trying to match the pattern and R.F find all will get all [01:01:13] the occurrences and organize them into a list now when you look at the um when you look at this pattern first of all notice that this is a raw string um and then these are three double quotes just to start the string so really the string itself this is the pattern itself right and notice that it's made up of a lot of ores so see these vertical bars those are ores in reg X and so you go from left to right in this pattern and try to match it against the string [01:01:43] wherever you are so we have hello and we're going to try to match it well it's not apostrophe s it's not apostrophe t or any of these but it is an optional space followed by- P of uh sorry SL P of L one or more times what is/ P of L it is coming to some documentation that I found um there might be other sources as well uh SLP is a letter any kind of letter from any language and hello is [01:02:15] made up of letters h e l Etc so optional space followed by a bunch of letters one or more letters is going to match hello but then the match ends because a white space is not a letter so from there on begins a new sort of attempt to match against the string again and starting in here we're going to skip over all of these again until we get to the exact same Point again and we see that there's an optional space this is the optional space followed by a bunch of letters one [01:02:46] or more of them and so that matches so when we run this we get a list of two elements hello and then space world so how are you if we add more letters we would just get them like this now what is this doing and why is this important we are taking our string and instead of directly encoding it um for tokenization we are first splitting it up and when you actually step through the code and we'll do that in a bit more detail what really is doing on a high [01:03:17] level is that it first splits your text into a list of texts just like this one and all these elements of this list are processed independently by the tokenizer and all of the results of that processing are simply concatenated so hello world oh I I missed how hello world how are you we have five elements of list all of these will independent independently go from text to a token [01:03:47] sequence and then that token sequence is going to be concatenated it's all going to be joined up and roughly speaking what that does is you're only ever finding merges between the elements of this list so you can only ever consider merges within every one of these elements in individually and um after you've done all the possible merging for all of these elements individually the results of all that will be joined um by concatenation and so you are basically what what you're doing effectively is [01:04:18] you are never going to be merging this e with this space because they are now parts of the separate elements of this list and so you are saying we are never going to merge eace um because we're breaking it up in this way so basically using this regx pattern to Chunk Up the text is just one way of enforcing that some merges are not to happen and we're going to go into more of this text and we'll see that what this is trying to do on a high level is we're trying to not merge [01:04:48] across letters across numbers across punctuation and so on so let's see in more detail how that works so let's continue now we have/ P ofn if you go to the documentation SLP of n is any kind of numeric character in any script so it's numbers so we have an optional space followed by numbers and those would be separated out so letters and numbers are being separated so if I do Hello World 123 how are you then world will stop matching here because one is not a letter anymore but one is a number [01:05:20] so this group will match for that and we'll get it as a separate entity uh let's see how these apostrophes work so here if we have um uh Slash V or I mean apostrophe V as an example then apostrophe here is not a letter or a number so hello will stop matching and then we will exactly match this with that so that will come out as a separate thing so why are they doing the [01:05:50] apostrophes here honestly I think that these are just like very common apostrophes p uh that are used um typically I don't love that they've done this because uh let me show you what happens when you have uh some Unicode apostrophes like for example you can have if you have house then this will be separated out because of this matching but if you use the Unicode apostrophe like this then suddenly this does not work and so this apostrophe will actually [01:06:21] become its own thing now and so so um it's basically hardcoded for this specific kind of apostrophe and uh otherwise they become completely separate tokens in addition to this you can go to the gpt2 docs and here when they Define the pattern they say should have added re. ignore case so BP merges can happen for capitalized versions of contractions so what they're pointing out is that you see how this is apostrophe and then lowercase letters well because they didn't do re. ignore [01:06:52] case then then um these rules will not separate out the apostrophes if it's uppercase so house would be like this but if I did house if I'm uppercase then notice suddenly the apostrophe comes by itself so the tokenization will work differently in uppercase and lower case inconsistently separating out these apostrophes so it feels extremely gnarly and slightly gross um but that's that's [01:07:24] how that works okay so let's come back after trying to match a bunch of apostrophe Expressions by the way the other issue here is that these are quite language specific probably so I don't know that all the languages for example use or don't use apostrophes but that would be inconsistently tokenized as a result then we try to match letters then we try to match numbers and then if that doesn't work we fall back to here and what this is saying is again optional space followed by something that is not a letter number or a space in one or more of that so what this is doing [01:07:55] effectively is this is trying to match punctuation roughly speaking not letters and not numbers so this group will try to trigger for that so if I do something like this then these parts here are not letters or numbers but they will actually they are uh they will actually get caught here and so they become its own group so we've separated out the punctuation and finally this um this is also a little bit confusing so this is matching white space but this is using a [01:08:25] negative look ahead assertion in regex so what this is doing is it's matching wh space up to but not including the last Whit space character why is this important um this is pretty subtle I think so you see how the white space is always included at the beginning of the word so um space r space u Etc suppose we have a lot of spaces here what's going to happen here is that these spaces up to not including the last character will get caught by this [01:08:57] and what that will do is it will separate out the spaces up to but not including the last character so that the last character can come here and join with the um space you and the reason that's nice is because space you is the common token so if I didn't have these Extra Spaces here you would just have space you and if I add tokens if I add spaces we still have a space view but now we have all this extra white space so basically the GB to tokenizer really likes to have a space letters or numbers [01:09:27] um and it it preens these spaces and this is just something that it is consistent about so that's what that is for and then finally we have all the the last fallback is um whites space characters uh so um that would be just um if that doesn't get caught then this thing will catch any trailing spaces and so on I wanted to show one more real world example here so if we have this string which is a piece of python code and then we try to split it up then this is the kind of output we [01:09:58] get so you'll notice that the list has many elements here and that's because we are splitting up fairly often uh every time sort of a category changes um so there will never be any merges Within These elements and um that's what you are seeing here now you might think that in order to train the tokenizer uh open AI has used this to split up text into chunks and then run just a BP algorithm within all the chunks but that is not exactly what happened and the reason is the following [01:10:30] notice that we have the spaces here uh those Spaces end up being entire elements but these spaces never actually end up being merged by by open Ai and the way you can tell is that if you copy paste the exact same chunk here into Tik token U Tik tokenizer you see that all the spaces are kept independent and they're all token 220 so I think opena at some point Point en Force some rule that these spaces would never be merged and so um there's some additional rules on top of just [01:11:01] chunking and bpe that open ey is not uh clear about now the training code for the gpt2 tokenizer was never released so all we have is uh the code that I've already shown you but this code here that they've released is only the inference code for the tokens so this is not the training code you can't give it a piece of text and training tokenizer this is just the inference code which Tak takes the merges that we have up above and applies them to a new piece of text and so we don't know exactly how opening ey trained um train the [01:11:32] tokenizer but it wasn't as simple as chunk it up and BP it uh whatever it was next I wanted to introduce you to the Tik token library from openai which is the official library for tokenization from openai so this is Tik token bip install P to Tik token and then um you can do the tokenization in inference this is again not training code this is only inference code for tokenization um I wanted to show you how you would use it quite simple and [01:12:02] running this just gives us the gpt2 tokens or the GPT 4 tokens so this is the tokenizer use for GPT 4 and so in particular we see that the Whit space in gpt2 remains unmerged but in GPT 4 uh these Whit spaces merge as we also saw in this one where here they're all unmerged but if we go down to GPT 4 uh they become merged um now in the gp4 uh tokenizer they changed the regular expression that they use to [01:12:33] Chunk Up text so the way to see this is that if you come to your the Tik token uh library and then you go to this file Tik token X openi public this is where sort of like the definition of all these different tokenizers that openi maintains is and so uh necessarily to do the inference they had to publish some of the details about the strings so this is the string that we already saw for gpt2 it is slightly different but it is actually equivalent uh to what we discussed here so this pattern that we discussed is equivalent to this [01:13:04] pattern this one just executes a little bit faster so here you see a little bit of a slightly different definition but otherwise it's the same we're going to go into special tokens in a bit and then if you scroll down to CL 100k this is the GPT 4 tokenizer you see that the pattern has changed um and this is kind of like the main the major change in addition to a bunch of other special tokens which I'll go into in a bit again now some I'm not going to actually go into the full detail of the pattern change because honestly this is my [01:13:35] numbing uh I would just advise that you pull out chat GPT and the regex documentation and just step through it but really the major changes are number one you see this eye here that means that the um case sensitivity this is case insensitive match and so the comment that we saw earlier on oh we should have used re. uppercase uh basically we're now going to be matching these apostrophe s apostrophe D apostrophe M Etc uh we're going to be [01:14:06] matching them both in lowercase and in uppercase so that's fixed there's a bunch of different like handling of the whites space that I'm not going to go into the full details of and then one more thing here is you will notice that when they match the numbers they only match one to three numbers so so they will never merge numbers that are in low in more than three digits only up to three digits of numbers will ever be merged and uh that's one change that they made as well [01:14:36] to prevent uh tokens that are very very long number sequences uh but again we don't really know why they do any of this stuff uh because none of this is documented and uh it's just we just get the pattern so um yeah it is what it is but those are some of the changes that gp4 has made and of course the vocabulary size went from roughly 50k to roughly 100K the next thing I would like to do very briefly is to take you through the gpt2 encoder dopy that openi has released uh this is the file that I [01:15:07] already mentioned to you briefly now this file is uh fairly short and should be relatively understandable to you at this point um starting at the bottom here they are loading two files encoder Json and vocab bpe and they do some light processing on it and then they call this encoder object which is the tokenizer now if you'd like to inspect these two files which together constitute their saved tokenizer then you can do that with a piece of code like this um this is where you can download [01:15:39] these two files and you can inspect them if you'd like and what you will find is that this encoder as they call it in their code is exactly equivalent to our vocab so remember here where we have this vocab object which allowed us us to decode very efficiently and basically it took us from the integer to the byes uh for that integer so our vocab is exactly their encoder and then their vocab bpe confusingly is actually are merges so [01:16:11] their BP merges which is based on the data inside vocab bpe ends up being equivalent to our merges so uh basically they are saving and loading the two uh variables that for us are also critical the merges variable and the vocab variable using just these two variables you can represent a tokenizer and you can both do encoding and decoding once you've trained this tokenizer now the only thing that um is actually slightly confusing inside what [01:16:42] opening ey does here is that in addition to this encoder and a decoder they also have something called a bite encoder and a bite decoder and this is actually unfortunately just kind of a spirous implementation detail and isn't actually deep or interesting in any way so I'm going to skip the discussion of it but what opening ey does here for reasons that I don't fully understand is that not only have they this tokenizer which can encode and decode but they have a whole separate layer here in addition that is used serially with the tokenizer and so you [01:17:12] first do um bite encode and then encode and then you do decode and then bite decode so that's the loop and they are just stacked serial on top of each other and and it's not that interesting so I won't cover it and you can step through it if you'd like otherwise this file if you ignore the bite encoder and the bite decoder will be algorithmically very familiar with you and the meat of it here is the what they call bpe function and you should recognize this Loop here which is very similar to our own y Loop where they're trying to identify the [01:17:43] Byram uh a pair that they should be merging next and then here just like we had they have a for Loop trying to merge this pair uh so they will go over all of the sequence and they will merge the pair whenever they find it and they keep repeating that until they run out of possible merges in the in the text so that's the meat of this file and uh there's an encode and a decode function just like we have implemented it so long story short what I want you to take away at this point is that unfortunately it's a little bit of a messy code that they [01:18:13] have but algorithmically it is identical to what we've built up above and what we've built up above if you understand it is algorithmically what is necessary to actually build a BP to organizer train it and then both encode and decode the next topic I would like to turn to is that of special tokens so in addition to tokens that are coming from you know raw bytes and the BP merges we can insert all kinds of tokens that we are going to use to delimit different parts of the data or introduced to create a special structure of the token streams [01:18:44] so in uh if you look at this encoder object from open AIS gpd2 right here we mentioned this is very similar to our vocab you'll notice that the length of this is 50257 and as I mentioned it's mapping uh and it's inverted from the mapping of our vocab our vocab goes from integer to string and they go the other way around for no amazing reason um but the thing to note here is that this the mapping table here is [01:19:15] 50257 where does that number come from where what are the tokens as I mentioned there are 256 raw bite token tokens and then opena actually did 50,000 merges so those become the other tokens but this would have been 50256 so what is the 57th token and there is basically one special token and that one special token you can see is called end of text so this is a [01:19:47] special token and it's the very last token and this token is used to delimit documents ments in the training set so when we're creating the training data we have all these documents and we tokenize them and we get a stream of tokens those tokens only range from Z to 50256 and then in between those documents we put special end of text token and we insert that token in between documents and we are using this as a signal to the language model that [01:20:18] the document has ended and what follows is going to be unrelated to the document previously that said the language model has to learn this from data it it needs to learn that this token usually means that it should wipe its sort of memory of what came before and what came before this token is not actually informative to what comes next but we are expecting the language model to just like learn this but we're giving it the Special sort of the limiter of these documents we can go here to Tech tokenizer and um this the gpt2 tokenizer uh our code that [01:20:49] we've been playing with before so we can add here right hello world world how are you and we're getting different tokens but now you can see what if what happens if I put end of text you see how until I finished it these are all different tokens end of text still set different tokens and now when I finish it suddenly we get token 50256 and the reason this works is because this didn't actually go through the bpe merges instead the code that [01:21:21] actually outposted tokens has special case instructions for handling special tokens um we did not see these special instructions for handling special tokens in the encoder dopy it's absent there but if you go to Tech token Library which is uh implemented in Rust you will find all kinds of special case handling for these special tokens that you can register uh create adds to the vocabulary and then it looks for them and it uh whenever it sees these special tokens like this it will actually come [01:21:53] in and swap in that special token so these things are outside of the typical algorithm of uh B PA en coding so these special tokens are used pervasively uh not just in uh basically base language modeling of predicting the next token in the sequence but especially when it gets to later to the fine tuning stage and all of the chat uh gbt sort of aspects of it uh because we don't just want to Del limit documents we want to delimit entire conversations between an assistant and a user so if I refresh this sck tokenizer page the [01:22:24] default example that they have here is using not sort of base model encoders but ftuned model uh sort of tokenizers um so for example using the GPT 3.5 turbo scheme these here are all special tokens I am start I end Etc uh this is short for Imaginary mcore start by the way but you can see here that there's a sort of start and end of every single message and there can be many other other tokens lots of tokens um in use to [01:22:56] delimit these conversations and kind of keep track of the flow of the messages here now we can go back to the Tik token library and here when you scroll to the bottom they talk about how you can extend tick token and I can you can create basically you can Fork uh the um CL 100K base tokenizers in gp4 and for example you can extend it by adding more special tokens and these are totally up to you you can come up with any arbitrary tokens and add them with the new ID afterwards and the tikken library [01:23:26] will uh correctly swap them out uh when it sees this in the strings now we can also go back to this file which we've looked at previously and I mentioned that the gpt2 in Tik toen open I.P we have the vocabulary we have the pattern for splitting and then here we are registering the single special token in gpd2 which was the end of text token and we saw that it has this ID in GPT 4 when they defy this here you [01:23:56] see that the pattern has changed as we've discussed but also the special tokens have changed in this tokenizer so we of course have the end of text just like in gpd2 but we also see three sorry four additional tokens here Thim prefix middle and suffix what is fim fim is short for fill in the middle and if you'd like to learn more about this idea it comes from this paper um and I'm not going to go into detail in this video it's beyond this video and then there's one additional uh serve token here so [01:24:27] that's that encoding as well so it's very common basically to train a language model and then if you'd like uh you can add special tokens now when you add special tokens you of course have to um do some model surgery to the Transformer and all the parameters involved in that Transformer because you are basically adding an integer and you want to make sure that for example your embedding Matrix for the vocabulary tokens has to be extended by adding a row and typically this row would be initialized uh with small random numbers or something like that because we need [01:24:58] to have a vector that now stands for that token in addition to that you have to go to the final layer of the Transformer and you have to make sure that that projection at the very end into the classifier uh is extended by one as well so basically there's some model surgery involved that you have to couple with the tokenization changes if you are going to add special tokens but this is a very common operation that people do especially if they'd like to fine tune the model for example taking it from a base model to a chat model like chat GPT okay so at this point you should [01:25:29] have everything you need in order to build your own gp4 tokenizer now in the process of developing this lecture I've done that and I published the code under this repository MBP so MBP looks like this right now as I'm recording but uh the MBP repository will probably change quite a bit because I intend to continue working on it um in addition to the MBP repository I've published the this uh exercise progression that you can follow so if you go to exercise. MD here uh this is sort of me breaking up the task ahead of [01:26:01] you into four steps that sort of uh build up to what can be a gp4 tokenizer and so feel free to follow these steps exactly and follow a little bit of the guidance that I've laid out here and anytime you feel stuck just reference the MBP repository here so either the tests could be useful or the MBP repository itself I try to keep the code fairly clean and understandable and so um feel free to reference it whenever um you get stuck uh in addition to that basically [01:26:32] once you write it you should be able to reproduce this behavior from Tech token so getting the gb4 tokenizer you can take uh you can encode the string and you should get these tokens and then you can encode and decode the exact same string to recover it and in addition to all that you should be able to implement your own train function uh which Tik token Library does not provide it's it's again only inference code but you could write your own train MBP does it as well and that will allow you to train your own token vocabularies so here are some of the [01:27:02] code inside M be mean bpe uh shows the token vocabularies that you might obtain so on the left uh here we have the GPT 4 merges uh so the first 256 are raw individual bytes and then here I am visualizing the merges that gp4 performed during its training so the very first merge that gp4 did was merge two spaces into a single token for you know two spaces and that is a token 256 and so this is the order in which things [01:27:32] merged during gb4 training and this is the merge order that um we obtain in MBP by training a tokenizer and in this case I trained it on a Wikipedia page of Taylor Swift uh not because I'm a Swifty but because that is one of the longest um Wikipedia Pages apparently that's available but she is pretty cool and um what was I going to say yeah so you can compare these two uh vocabularies and so as an example um here GPT for [01:28:04] merged I in to become in and we've done the exact same thing on this token 259 here space t becomes space t and that happened for us a little bit later as well so the difference here is again to my understanding only a difference of the training set so as an example because I see a lot of white space I supect that gp4 probably had a lot of python code in its training set I'm not sure uh for the tokenizer and uh here we see much less of that of course in the Wikipedia page so roughly speaking they look the same [01:28:34] and they look the same because they're running the same algorithm and when you train your own you're probably going to get something similar depending on what you train it on okay so we are now going to move on from tick token and the way that open AI tokenizes its strings and we're going to discuss one more very commonly used library for working with tokenization inlm and that is sentence piece so sentence piece is very commonly used in language models because unlike Tik token it can do both training and inference and is quite efficient at both it supports a [01:29:04] number of algorithms for training uh vocabularies but one of them is the B pair en coding algorithm that we've been looking at so it supports it now sentence piece is used both by llama and mistal series and many other models as well it is on GitHub under Google sentence piece and the big difference with sentence piece and we're going to look at example because this is kind of hard and subtle to explain is that they think different about the order of operations here so in [01:29:35] the case of Tik token we first take our code points in the string we encode them using mutf to bytes and then we're merging bytes it's fairly straightforward for sentence piece um it works directly on the level of the code points themselves so so it looks at whatever code points are available in your training set and then it starts merging those code points and um the bpe is running on the level of code points and if you happen to run out of code points so there are maybe some rare [01:30:06] uh code points that just don't come up too often and the Rarity is determined by this character coverage hyper parameter then these uh code points will either get mapped to a special unknown token like ank or if you have the bite foldback option turned on then that will take those rare Cod points it will encode them using utf8 and then the individual bytes of that encoding will be translated into tokens and there are these special bite tokens that basically get added to the vocabulary so it uses BP on on the code points and then it [01:30:38] falls back to bytes for rare Cod points um and so that's kind of like difference personally I find the Tik token we significantly cleaner uh but it's kind of like a subtle but pretty major difference between the way they approach tokenization let's work with with a concrete example because otherwise this is kind of hard to um to get your head around so let's work with a concrete example this is how we can import sentence piece and then here we're going to take I think I took like the description of sentence piece and I just created like a little toy data set it [01:31:08] really likes to have a file so I created a toy. txt file with this content now what's kind of a little bit crazy about sentence piece is that there's a ton of options and configurations and the reason this is so is because sentence piece has been around I think for a while and it really tries to handle a large diversity of things and um because it's been around I think it has quite a bit of accumulated historical baggage uh as well and so in particular there's like a ton of configuration arguments this is not even all of it you can go to here to see all [01:31:39] the training options um and uh there's also quite useful documentation when you look at the raw Proto buff uh that is used to represent the trainer spec and so on um many of these options are irrelevant to us so maybe to point out one example Das Das shrinking Factor uh this shrinking factor is not used in the B pair en coding algorithm so this is just an argument that is irrelevant to us um it applies to a different training [01:32:09] algorithm now what I tried to do here is I tried to set up sentence piece in a way that is very very similar as far as I can tell to maybe identical hopefully to the way that llama 2 was strained so the way they trained their own um their own tokenizer and the way I did this was basically you can take the tokenizer model file that meta released and you can um open it using the Proto protuff uh sort of file that you can generate and then you can inspect all the options [01:32:39] and I tried to copy over all the options that looked relevant so here we set up the input it's raw text in this file here's going to be the output so it's going to be for talk 400. model and vocab we're saying that we're going to use the BP algorithm and we want to Bap size of 400 then there's a ton of configurations here for um for basically pre-processing and normalization rules as they're called normalization used to be very prevalent [01:33:09] I would say before llms in natural language processing so in machine translation and uh text classification and so on you want to normalize and simplify the text and you want to turn it all lowercase and you want to remove all double whites space Etc and in language models we prefer not to do any of it or at least that is my preference as a deep learning person you want to not touch your data you want to keep the raw data as much as possible um in a raw form so you're basically trying to turn off a lot of this if you can the other thing that sentence piece does is that [01:33:39] it has this concept of sentences so sentence piece it's back it's kind of like was developed I think early in the days where there was um an idea that they you're training a tokenizer on a bunch of independent sentences so it has a lot of like how many sentences you're going to train on what is the maximum sentence length um shuffling sentences and so for it sentences are kind of like the individual training examples but again in the context of llms I find that this is like a very spous and weird [01:34:10] distinction like sentences are just like don't touch the raw data sentences happen to exist but in raw data sets there are a lot of like inet like what exactly is a sentence what isn't a sentence um and so I think like it's really hard to Define what an actual sentence is if you really like dig into it and there could be different concepts of it in different languages or something like that so why even introduce the concept it it doesn't honestly make sense to me I would just prefer to treat a file as a giant uh stream of [01:34:40] bytes it has a lot of treatment around rare word characters and when I say word I mean code points we're going to come back to this in a second and it has a lot of other rules for um basically splitting digits splitting white space and numbers and how you deal with that so these are some kind of like merge rules so I think this is a little bit equivalent to tick token using the regular expression to split up categories there's like kind of equivalence of it if you squint T it in sentence piece where you can also for [01:35:10] example split up split up the digits uh and uh so on there's a few more things here that I'll come back to in a bit and then there are some special tokens that you can indicate and it hardcodes the UN token the beginning of sentence end of sentence and a pad token um and the UN token must exist for my understanding and then some some things so we can train and when when I press train it's going to create this file talk 400. [01:35:40] model and talk 400. wab I can then load the model file and I can inspect the vocabulary off it and so we trained vocab size 400 on this text here and these are the individual pieces the individual tokens that sentence piece will create so in the beginning we see that we have the an token uh with the ID zero then we have the beginning of sequence end of sequence one and two and then we said that the pad ID is negative 1 so we chose not to use it so there's [01:36:12] no pad ID here then these are individual bite tokens so here we saw that bite fallback in llama was turned on so it's true so what follows are going to be the 256 bite tokens and these are their IDs and then at the bottom after the bite tokens come the merges and these are the parent nodes in the merges so we're not seeing the [01:36:42] children we're just seeing the parents and their ID and then after the merges comes eventually the individual tokens and their IDs and so these are the individual tokens so these are the individual code Point tokens if you will and they come at the end so that is the ordering with which sentence piece sort of like represents its vocabularies it starts with special tokens then the bike tokens then the merge tokens and then the individual codo tokens and all these raw codepoint to tokens are the ones [01:37:14] that it encountered in the training set so those individual code points are all the the entire set of code points that occurred here so those all get put in there and then those that are extremely rare as determined by character coverage so if a code Point occurred only a single time out of like a million um sentences or something like that then it would be ignored and it would not be added to our uh vocabulary once we have a vocabulary we can encode into IDs and we can um sort [01:37:46] of get a list and then here I am also decoding the indiv idual tokens back into little pieces as they call it so let's take a look at what happened here hello space on so these are the token IDs we got back and when we look here uh a few things sort of uh jump to mind number one take a look at these characters the Korean characters of course were not part of the training set so sentence [01:38:18] piece is encountering code points that it has not seen during training time and those code points do not have a token associated with them so suddenly these are un tokens unknown tokens but because bite fall back as true instead sentence piece falls back to bytes and so it takes this it encodes it with utf8 and then it uses these tokens to represent uh those bytes and that's what we are getting sort of here this is the utf8 uh [01:38:49] encoding and in this shifted by three uh because of these um special tokens here that have IDs earlier on so that's what happened here now one more thing that um well first before I go on with respect to the bitef back let me remove bite foldback if this is false what's going to happen let's retrain so the first thing that happened is all the bite tokens disappeared right and now we just have the merges and we [01:39:19] have a lot more merges now because we have a lot more space because we're not taking up space in the wab size uh with all the bytes and now if we encode this we get a zero so this entire string here suddenly there's no bitef back so this is unknown and unknown is an and so this is zero because the an token is token zero and you have to keep in mind that this would feed into your uh language model so what is a language model supposed to do when all kinds of [01:39:49] different things that are unrecognized because they're rare just end up mapping into Unk it's not exactly the property that you want so that's why I think llama correctly uh used by fallback true uh because we definitely want to feed these um unknown or rare code points into the model and some uh some manner the next thing I want to show you is the following notice here when we are decoding all the individual tokens you see how spaces uh space here ends up being this um bold underline I'm not [01:40:21] 100% sure by the way why sentence piece switches whites space into these bold underscore characters maybe it's for visualization I'm not 100% sure why that happens uh but notice this why do we have an extra space in the front of hello um what where is this coming from well it's coming from this option here um add dummy prefix is true and when you go to the documentation add D whites space at the [01:40:51] beginning of text in order to treat World in world and hello world in the exact same way so what this is trying to do is the following if we go back to our tick tokenizer world as uh token by itself has a different ID than space world so we have this is 1917 but this is 14 Etc so these are two different tokens for the language model and the language model has to learn from data that they are actually kind of like a very similar concept so to the language model in the [01:41:23] Tik token World um basically words in the beginning of sentences and words in the middle of sentences actually look completely different um and it has to learned that they are roughly the same so this add dami prefix is trying to fight that a little bit and the way that works is that it basically uh adds a dummy prefix so for as a as a part of pre-processing it will take the string and it will add a space it will do this and that's done in an effort to [01:41:54] make this world and that world the same they will both be space world so that's one other kind of pre-processing option that is turned on and llama 2 also uh uses this option and that's I think everything that I want to say for my preview of sentence piece and how it is different um maybe here what I've done is I just uh put in the Raw protocol buffer representation basically of the tokenizer the too trained so feel free to sort of Step through this and if you [01:42:24] would like uh your tokenization to look identical to that of the meta uh llama 2 then you would be copy pasting these settings as I tried to do up above and uh yeah that's I think that's it for this section I think my summary for sentence piece from all of this is number one I think that there's a lot of historical baggage in sentence piece a lot of Concepts that I think are slightly confusing and I think potentially um contain foot guns like this concept of a sentence and it's maximum length and stuff like that um otherwise it is fairly commonly used in [01:42:55] the industry um because it is efficient and can do both training and inference uh it has a few quirks like for example un token must exist and the way the bite fallbacks are done and so on I don't find particularly elegant and unfortunately I have to say it's not very well documented so it took me a lot of time working with this myself um and just visualizing things and trying to really understand what is happening here because uh the documentation unfortunately is in my opion not not super amazing but it is a very nice repo that is available to you if you'd like [01:43:26] to train your own tokenizer right now okay let me now switch gears again as we're starting to slowly wrap up here I want to revisit this issue in a bit more detail of how we should set the vocap size and what are some of the considerations around it so for this I'd like to go back to the model architecture that we developed in the last video when we built the GPT from scratch so this here was uh the file that we built in the previous video and we defined the Transformer model and and let's specifically look at Bap size and where it appears in this file so here we Define the voap size uh at this time it [01:43:58] was 65 or something like that extremely small number so this will grow much larger you'll see that Bap size doesn't come up too much in most of these layers the only place that it comes up to is in exactly these two places here so when we Define the language model there's the token embedding table which is this two-dimensional array where the vocap size is basically the number of rows and uh each vocabulary element each token has a vector that we're going to train using back propagation that Vector is of size and embed which is number of [01:44:29] channels in the Transformer and basically as voap size increases this embedding table as I mentioned earlier is going to also grow we're going to be adding rows in addition to that at the end of the Transformer there's this LM head layer which is a linear layer and you'll notice that that layer is used at the very end to produce the logits uh which become the probabilities for the next token in sequence and so intuitively we're trying to produce a probability for every single token that might come next at every point in time of that Transformer and if we have more [01:45:01] and more tokens we need to produce more and more probabilities so every single token is going to introduce an additional dot product that we have to do here in this linear layer for this final layer in a Transformer so why can't vocap size be infinite why can't we grow to Infinity well number one your token embedding table is going to grow uh your linear layer is going to grow so we're going to be doing a lot more computation here because this LM head layer will become more computational expensive number two because we have more parameters we could be worried that we are going to be under [01:45:33] trining some of these parameters so intuitively if you have a very large vocabulary size say we have a million uh tokens then every one of these tokens is going to come up more and more rarely in the training data because there's a lot more other tokens all over the place and so we're going to be seeing fewer and fewer examples uh for each individual token and you might be worried that basically the vectors associated with every token will be undertrained as a result because they just don't come up too often and they don't participate in the forward backward pass in addition to that as [01:46:03] your vocab size grows you're going to start shrinking your sequences a lot right and that's really nice because that means that we're going to be attending to more and more text so that's nice but also you might be worrying that two large of chunks are being squished into single tokens and so the model just doesn't have as much of time to think per sort of um some number of characters in the text or you can think about it that way right so basically we're squishing too much information into a single token and then the forward pass of the Transformer is [01:46:33] not enough to actually process that information appropriately and so these are some of the considerations you're thinking about when you're designing the vocab size as I mentioned this is mostly an empirical hyperparameter and it seems like in state-of-the-art architectures today this is usually in the high 10,000 or somewhere around 100,000 today and the next consideration I want to briefly talk about is what if we want to take a pre-trained model and we want to extend the vocap size and this is done fairly commonly actually so for example when you're doing fine-tuning for cha GPT um a lot more new special tokens get [01:47:03] introduced on top of the base model to maintain the metadata and all the structure of conversation objects between a user and an assistant so that takes a lot of special tokens you might also try to throw in more special tokens for example for using the browser or any other tool and so it's very tempting to add a lot of tokens for all kinds of special functionality so if you want to be adding a token that's totally possible Right all we have to do is we have to resize this embedding so we have to add rows we would initialize these uh parameters from scratch to be small [01:47:34] random numbers and then we have to extend the weight inside this linear uh so we have to start making dot products um with the associated parameters as well to basically calculate the probabilities for these new tokens so both of these are just a resizing operation it's a very mild model surgery and can be done fairly easily and it's quite common that basically you would freeze the base model you introduce these new parameters and then you only train these new parameters to introduce new tokens into the architecture um and so you can freeze arbitrary parts of it or you can [01:48:04] train arbitrary parts of it and that's totally up to you but basically minor surgery required if you'd like to introduce new tokens and finally I'd like to mention that actually there's an entire design space of applications in terms of introducing new tokens into a vocabulary that go Way Beyond just adding special tokens and special new functionality so just to give you a sense of the design space but this could be an entire video just by itself uh this is a paper on learning to compress prompts with what they called uh gist tokens and the rough idea is suppose that you're using language models in a [01:48:34] setting that requires very long prompts while these long prompts just slow everything down because you have to encode them and then you have to use them and then you're tending over them and it's just um you know heavy to have very large prompts so instead what they do here in this paper is they introduce new tokens and um imagine basically having a few new tokens you put them in a sequence and then you train the model by distillation so you are keeping the entire model Frozen and you're only training the representations of the new [01:49:05] tokens their embeddings and you're optimizing over the new tokens such that the behavior of the language model is identical uh to the model that has a very long prompt that works for you and so it's a compression technique of compressing that very long prompt into those few new gist tokens and so you can train this and then at test time you can discard your old prompt and just swap in those tokens and they sort of like uh stand in for that very long prompt and have an almost identical performance and so this is one um technique and a class [01:49:36] of parameter efficient fine-tuning techniques where most of the model is basically fixed and there's no training of the model weights there's no training of Laura or anything like that of new parameters the the parameters that you're training are now just the uh token embeddings so that's just one example but this could again be like an entire video but just to give you a sense that there's a whole design space here that is potentially worth exploring in the future the next thing I want to briefly address is that I think recently there's a lot of momentum in how you actually could construct Transformers that can simultaneously process not just [01:50:06] text as the input modality but a lot of other modalities so be it images videos audio Etc and how do you feed in all these modalities and potentially predict these modalities from a Transformer uh do you have to change the architecture in some fundamental way and I think what a lot of people are starting to converge towards is that you're not changing the architecture you stick with the Transformer you just kind of tokenize your input domains and then call the day and pretend it's just text tokens and just do everything else identical in an identical manner so here for example [01:50:36] there was a early paper that has nice graphic for how you can take an image and you can chunc at it into integers um and these sometimes uh so these will basically become the tokens of images as an example and uh these tokens can be uh hard tokens where you force them to be integers they can also be soft tokens where you uh sort of don't require uh these to be discrete but you do Force these representations to go through bottlenecks like in Auto encoders uh also in this paper that came [01:51:06] out from open a SORA which I think really um uh blew the mind of many people and inspired a lot of people in terms of what's possible they have a Graphic here and they talk briefly about how llms have text tokens Sora has visual patches so again they came up with a way to chunc a videos into basically tokens when they own vocabularies and then you can either process discrete tokens say with autog regressive models or even soft tokens with diffusion models and uh all of that is sort of uh being actively worked on [01:51:38] designed on and is beyond the scope of this video but just something I wanted to mention briefly okay now that we have come quite deep into the tokenization algorithm and we understand a lot more about how it works let's loop back around to the beginning of this video and go through some of these bullet points and really see why they happen so first of all why can't my llm spell words very well or do other spell related tasks so fundamentally this is because as we saw these characters are chunked up into tokens and some of these tokens are actually fairly long so as an [01:52:10] example I went to the gp4 vocabulary and I looked at uh one of the longer tokens so that default style turns out to be a single individual token so that's a lot of characters for a single token so my suspicion is that there's just too much crammed into this single token and my suspicion was that the model should not be very good at tasks related to spelling of this uh single token so I asked how many letters L are there in the word default style and of course my [01:52:41] prompt is intentionally done that way and you see how default style will be a single token so this is what the model sees so my suspicion is that it wouldn't be very good at this and indeed it is not it doesn't actually know how many L's are in there it thinks there are three and actually there are four if I'm not getting this wrong myself so that didn't go extremely well let's look look at another kind of uh character level task so for example here I asked uh gp4 to reverse the string default style and [01:53:11] they tried to use a code interpreter and I stopped it and I said just do it just try it and uh it gave me jumble so it doesn't actually really know how to reverse this string going from right to left uh so it gave a wrong result so again like working with this working hypothesis that maybe this is due to the tokenization I tried a different approach I said okay let's reverse the exact same string but take the following approach step one just print out every single character separated by spaces and then as a step two reverse that list and [01:53:43] it again Tred to use a tool but when I stopped it it uh first uh produced all the characters and that was actually correct and then It reversed them and that was correct once it had this so somehow it can't reverse it directly but when you go just first uh you know listing it out in order it can do that somehow and then it can once it's uh broken up this way this becomes all these individual characters and so now this is much easier for it to see these individual tokens and reverse them and print them out so that is kind of [01:54:13] interesting so let's continue now why are llms worse at uh non-english langu and I briefly covered this already but basically um it's not only that the language model sees less non-english data during training of the model parameters but also the tokenizer is not um is not sufficiently trained on non-english data and so here for example hello how are you is five tokens and its translation is 15 tokens so this is a three times blow up and so for example [01:54:45] anang is uh just hello basically in Korean and that end up being three tokens I'm actually kind of surprised by that because that is a very common phrase there just the typical greeting of like hello and that ends up being three tokens whereas our hello is a single token and so basically everything is a lot more bloated and diffuse and this is I think partly the reason that the model Works worse on other languages uh coming back why is LM bad at simple arithmetic um that has to do with the tokenization of numbers and so [01:55:17] um you'll notice that for example addition is very sort of like uh there's an algorithm that is like character level for doing addition so for example here we would first add the ones and then the tens and then the hundreds you have to refer to specific parts of these digits but uh these numbers are represented completely arbitrarily based on whatever happened to merge or not merge during the tokenization process there's an entire blog post about this that I think is quite good integer tokenization is insane and this person basically systematically explores the tokenization [01:55:48] of numbers in I believe this is gpt2 and so they notice that for example for the for um four-digit numbers you can take a look at whether it is uh a single token or whether it is two tokens that is a 1 three or a 2 two or a 31 combination and so all the different numbers are all the different combinations and you can imagine this is all completely arbitrarily so and the model unfortunately sometimes sees uh four um a token for for all four digits sometimes for three sometimes for two [01:56:18] sometimes for one and it's in an arbitrary uh Manner and so this is definitely a headwind if you will for the language model and it's kind of incredible that it can kind of do it and deal with it but it's also kind of not ideal and so that's why for example we saw that meta when they train the Llama 2 algorithm and they use sentence piece they make sure to split up all the um all the digits as an example for uh llama 2 and this is partly to improve a simple arithmetic kind of performance and finally why is gpt2 not [01:56:50] as good in Python again this is partly a modeling issue on in the architecture and the data set and the strength of the model but it's also partially tokenization because as we saw here with the simple python example the encoding efficiency of the tokenizer for handling spaces in Python is terrible and every single space is an individual token and this dramatically reduces the context length that the model can attend to cross so that's almost like a tokenization bug for gpd2 and that was later fixed with gp4 okay so here's [01:57:20] another fun one my llm abruptly halts when it sees the string end of text so here's um here's a very strange Behavior print a string end of text is what I told jt4 and it says could you please specify the string and I'm I'm telling it give me end of text and it seems like there's an issue it's not seeing end of text and then I give it end of text is the string and then here's a string and then it just doesn't print it so obviously something is breaking here with respect to the handling of the special token and I don't actually know [01:57:50] what open ey is doing under the hood here and whether they are potentially parsing this as an um as an actual token instead of this just being uh end of text um as like individual sort of pieces of it without the special token handling logic and so it might be that someone when they're calling do encode uh they are passing in the allowed special and they are allowing end of text as a special character in the user prompt but the user prompt of course is [01:58:20] is a sort of um attacker controlled text so you would hope that they don't really parse or use special tokens or you know from that kind of input but it appears that there's something definitely going wrong here and um so your knowledge of these special tokens ends up being in a tax surface potentially and so if you'd like to confuse llms then just um try to give them some special tokens and see if you're breaking something by chance okay so this next one is a really fun one uh the trailing whites space issue so if [01:58:52] you come to playground and uh we come here to GPT 3.5 turbo instruct so this is not a chat model this is a completion model so think of it more like it's a lot more closer to a base model it does completion it will continue the token sequence so here's a tagline for ice cream shop and we want to continue the sequence and so we can submit and get a bunch of tokens okay no problem but now suppose I do this but instead of pressing submit here I do here's a [01:59:23] tagline for ice cream shop space so I have a space here before I click submit we get a warning your text ends in a trail Ling space which causes worse performance due to how API splits text into tokens so what's happening here it still gave us a uh sort of completion here but let's take a look at what's happening so here's a tagline for an ice cream shop and then what does this look like in the actual actual training data suppose you found the completion in the training document somewhere on the [01:59:53] internet and the llm trained on this data so maybe it's something like oh yeah maybe that's the tagline that's a terrible tagline but notice here that when I create o you see that because there's the the space character is always a prefix to these tokens in GPT so it's not an O token it's a space o token the space is part of the O and together they are token 8840 that's that's space o so what's What's Happening Here is that when I just have [02:00:24] it like this and I let it complete the next token it can sample the space o token but instead if I have this and I add my space then what I'm doing here when I incode this string is I have basically here's a t line for an ice cream uh shop and this space at the very end becomes a token 220 and so we've added token 220 and this token otherwise would be part of the tagline because if there actually is a tagline here so space o is the token [02:00:55] and so this is suddenly a of distribution for the model because this space is part of the next token but we're putting it here like this and the model has seen very very little data of actual Space by itself and we're asking it to complete the sequence like add in more tokens but the problem is that we've sort of begun the first token and now it's been split up and now we're out of this distribution and now arbitrary bad things happen and it's just a very rare example for it to see something like that and uh that's why we get the [02:01:26] warning so the fundamental issue here is of course that um the llm is on top of these tokens and these tokens are text chunks they're not characters in a way you and I would think of them they are these are the atoms of what the LM is seeing and there's a bunch of weird stuff that comes out of it let's go back to our default cell style I bet you that the model has never in its training set seen default cell sta without Le in there it's always seen this as a single [02:01:56] group because uh this is some kind of a function in um I'm guess I don't actually know what this is part of this is some kind of API but I bet you that it's never seen this combination of tokens uh in its training data because or I think it would be extremely rare so I took this and I copy pasted it here and I had I tried to complete from it and the it immediately gave me a big error and it said the model predicted to completion that begins with a stop sequence resulting in no output consider adjusting your prompt or stop sequences [02:02:26] so what happened here when I clicked submit is that immediately the model emitted and sort of like end of text token I think or something like that it basically predicted the stop sequence immediately so it had no completion and so this is why I'm getting a warning again because we're off the data distribution and the model is just uh predicting just totally arbitrary things it's just really confused basically this is uh this is giving it brain damage it's never seen this before it's shocked and it's predicting end of text or something I tried it again here and it [02:02:57] in this case it completed it but then for some reason this request May violate our usage policies this was flagged um basically something just like goes wrong and there's something like Jank you can just feel the Jank because the model is like extremely unhappy with just this and it doesn't know how to complete it because it's never occurred in training set in a training set it always appears like this and becomes a single token so these kinds of issues where tokens are either you sort of like complete the first character of the next token or you are sort of you have long tokens that [02:03:28] you then have just some of the characters off all of these are kind of like issues with partial tokens is how I would describe it and if you actually dig into the T token repository go to the rust code and search for unstable and you'll see um en code unstable native unstable token tokens and a lot of like special case handling none of this stuff about unstable tokens is documented anywhere but there's a ton of code dealing with unstable tokens and [02:03:58] unstable tokens is exactly kind of like what I'm describing here what you would like out of a completion API is something a lot more fancy like if we're putting in default cell sta if we're asking for the next token sequence we're not actually trying to append the next token exactly after this list we're actually trying to append we're trying to consider lots of tokens um that if we were or I guess like we're trying to search over characters that if we retened would be of high probability [02:04:28] if that makes sense um so that we can actually add a single individual character uh instead of just like adding the next full token that comes after this partial token list so I this is very tricky to describe and I invite you to maybe like look through this it ends up being extremely gnarly and hairy kind of topic it and it comes from tokenization fundamentally so um maybe I can even spend an entire video talking about unstable tokens sometime in the future okay and I'm really saving the best for last my favorite one by far is the solid gold [02:04:59] Magikarp and it just okay so this comes from this blog post uh solid gold Magikarp and uh this is um internet famous now for those of us in llms and basically I I would advise you to uh read this block Post in full but basically what this person was doing is this person went to the um token embedding stable and clustered the tokens based on their embedding representation and this person noticed that there's a cluster of tokens that [02:05:29] look really strange so there's a cluster here at rot e stream Fame solid gold Magikarp Signet message like really weird tokens in uh basically in this embedding cluster and so what are these tokens and where do they even come from like what is solid gold magikarpet makes no sense and then they found bunch of these tokens and then they notice that actually the plot thickens here because if you ask the model about these tokens like you ask it uh some very benign question like please can you repeat back [02:06:00] to me the string sold gold Magikarp uh then you get a variety of basically totally broken llm Behavior so either you get evasion so I'm sorry I can't hear you or you get a bunch of hallucinations as a response um you can even get back like insults so you ask it uh about streamer bot it uh tells the and the model actually just calls you names uh or it kind of comes up with like weird humor like you're actually breaking the model by asking about these very simple strings like at Roth and [02:06:30] sold gold Magikarp so like what the hell is happening and there's a variety of here documented behaviors uh there's a bunch of tokens not just so good Magikarp that have that kind of a behavior and so basically there's a bunch of like trigger words and if you ask the model about these trigger words or you just include them in your prompt the model goes haywire and has all kinds of uh really Strange Behaviors including sort of ones that violate typical safety guidelines uh and the alignment of the model like it's swearing back at you so what is happening here and how can this [02:07:01] possibly be true well this again comes down to tokenization so what's happening here is that sold gold Magikarp if you actually dig into it is a Reddit user so there's a u Sol gold Magikarp and probably what happened here even though I I don't know that this has been like really definitively explored but what is thought to have happened is that the tokenization data set was very different from the training data set for the actual language model so in the tokenization data set there was a ton of [02:07:31] redded data potentially where the user solid gold Magikarp was mentioned in the text because solid gold Magikarp was a very common um sort of uh person who would post a lot uh this would be a string that occurs many times in a tokenization data set because it occurs many times in a tokenization data set these tokens would end up getting merged to the single individual token for that single Reddit user sold gold Magikarp so they would have a dedicated token in a vocabulary of was it 50,000 tokens in gpd2 that is devoted to that Reddit user [02:08:04] and then what happens is the tokenization data set has those strings but then later when you train the model the language model itself um this data from Reddit was not present and so therefore in the entire training set for the language model sold gold Magikarp never occurs that token never appears in the training set for the actual language model later so this token never gets activated it's initialized at random in the beginning of optimization then you have forward backward passes and updates [02:08:34] to the model and this token is just never updated in the embedding table that row Vector never gets sampled it never gets used so it never gets trained and it's completely untrained it's kind of like unallocated memory in a typical binary program written in C or something like that that so it's unallocated memory and then at test time if you evoke this token then you're basically plucking out a row of the embedding table that is completely untrained and that feeds into a Transformer and creates undefined behavior and that's what we're seeing here this completely undefined never before seen in a training behavior and so any of these [02:09:06] kind of like weird tokens would evoke this Behavior because fundamentally the model is um is uh uh out of sample out of distribution okay and the very last thing I wanted to just briefly mention point out although I think a lot of people are quite aware of this is that different kinds of formats and different representations and different languages and so on might be more or less efficient with GPD tokenizers uh or any tokenizers for any other L for that matter so for example Json is actually really dense in tokens and yaml is a lot [02:09:36] more efficient in tokens um so for example this are these are the same in Json and in yaml the Json is 116 and the yaml is 99 so quite a bit of an Improvement and so in the token economy where we are paying uh per token in many ways and you are paying in the context length and you're paying in um dollar amount for uh the cost of processing all this kind of structured data when you have to um so prefer to use theal over Json and in general kind [02:10:06] of like the tokenization density is something that you have to um sort of care about and worry about at all times and try to find efficient encoding schemes and spend a lot of time in tick tokenizer and measure the different token efficiencies of different formats and settings and so on okay so that concludes my fairly long video on tokenization I know it's a try I know it's annoying I know it's irritating I personally really dislike the stage what I do have to say at this point is don't brush it off there's a lot of foot guns sharp edges here security issues uh AI [02:10:38] safety issues as we saw plugging in unallocated memory into uh language models so um it's worth understanding this stage um that said I will say that eternal glory goes to anyone who can get rid of it uh I showed you one possible paper that tried to uh do that and I think I hope a lot more can follow over time and my final recommendations for the application right now are if you can reuse the GPT 4 tokens and the vocabulary uh in your application then that's something you should consider and just use Tech token because it is very efficient and nice library for inference [02:11:11] for bpe I also really like the bite level BP that uh Tik toen and openi uses uh if you for some reason want to train your own vocabulary from scratch um then I would use uh the bpe with sentence piece um oops as I mentioned I'm not a huge fan of sentence piece I don't like its uh bite fallback and I don't like that it's doing BP on unic code code points I think it's uh it also has like a million settings and I think there's a lot of foot gonss here and I think it's really easy to Mis calibrate them and [02:11:42] you end up cropping your sentences or something like that uh because of some type of parameter that you don't fully understand so so be very careful with the settings try to copy paste exactly maybe where what meta did or basically spend a lot of time looking at all the hyper parameters and go through the code of sentence piece and make sure that you have this correct um but even if you have all the settings correct I still think that the algorithm is kind of inferior to what's happening here and maybe the best if you really need to train your vocabulary maybe the best thing is to just wait for M bpe to [02:12:13] becomes as efficient as possible and uh that's something that maybe I hope to work on and at some point maybe we can be training basically really what we want is we want tick token but training code and that is the ideal thing that currently does not exist and MBP is um is in implementation of it but currently it's in Python so that's currently what I have to say for uh tokenization there might be an advanced video that has even drier and even more detailed in the future but for now I think we're going [02:12:43] to leave things off here and uh I hope that was helpful bye and uh they increase this contact size from gpt1 of 512 uh to 1024 and GPT 4 two the next okay next I would like us to briefly walk through the code from open AI on the gpt2 encoded [02:13:15] ATP I'm sorry I'm gonna sneeze and then what's Happening Here is this is a spous layer that I will explain in a bit What's Happening Here is
Keep going
3:31:23
Andrej Karpathy

Deep Dive into LLMs like ChatGPT

Andrej Karpathy's general audience deep dive into the full training stack behind ChatGPT, in one sitting: pretraining on a filtered crawl of the internet, tokenization, what the base model actually is (an internet document simulator), supervised fine tuning that turns it into an assistant, and reinforcement learning as the third and least mature stage. The second half is LLM psychology: why hallucinations happen and how they get patched, why models need tokens to think, why they cannot spell or compare 9.11 to 9.9, and what emergent chains of thought in DeepSeek R1 mean. It ends with where to find and run these models yourself.

AIDeep LearningFeb 5, 2025
4:01:26
Andrej Karpathy

Let's reproduce GPT-2 (124M)

Four hours, empty file to trained model. Karpathy writes the GPT-2 124M architecture to match the released weights exactly, then spends most of the video making it fast: TF32 and bfloat16, torch.compile, flash attention, choosing vocabulary sizes with nice powers of two, then the optimization recipe from the GPT-3 paper (AdamW settings, gradient clipping, cosine schedule with warmup, weight decay, gradient accumulation to reach a half million token batch) and finally distributed training across eight GPUs. It ends with a real run on FineWeb-Edu that beats the published GPT-2 124M numbers.

AIDeep LearningJun 9, 2024
26:15
Brandon Butch

macOS 27 Golden Gate - 50+ New Features & Changes!

Brandon Butch walks through more than fifty changes in macOS 27 Golden Gate beta 1. Two stories run through it: a design walkback that fixes Tahoe's loudest complaints (corner radii, floating sidebars, toolbars), and an AI first turn where Spotlight and Siri merge into one Search or Ask box, Siri gains on screen awareness, personal context, and in app actions, and vibe coding arrives for Safari extensions and Shortcuts. This page rebuilds the whole tour, feature by feature, with the live demos and the honest beta misses.

AppleAIJun 14, 2026