Skip to main content
PProfileMatch
Read Play
← Path

Large Language Models · Large Language Models: Tokens and Context Windows Explained

0 XP
  1. 1 current stop: learning
  2. 2 upcoming
  3. 3 upcoming
  4. 4 upcoming
  5. 5 upcoming

Finish after stop 5

Stop 1 of 5 · Bike shop: sorting the spare parts bin

Trick to keep in mind+3 XP

Lego bricks: a small set of pieces can build any model.

A tokenizer has a fixed list of pieces, from whole common words down to single characters. Because small pieces are always available, any text can be spelled out, even a word the model never met whole. The price is that unusual words break into more pieces, and every piece is one more token to pay for.

Read it once, then a question tests whether the trick stuck.