#ai

2 public notes

Why a language model processes tokens rather than whole words

Language-model text is commonly divided into tokens, which may represent words, word parts, punctuation or other sequences. The tokenizer determines that division before the model processes the text. A sentence's token count therefore depends on the encoding scheme and language, not simply the number of spaces.

Technology♡ 0🔖 0

ai

What changes when a neural network is trained

Training a neural network adjusts numerical parameters so its outputs better match a chosen objective on training data. An error measure guides those adjustments through repeated examples. The process can produce useful patterns, but good training performance alone does not show that the model will handle unfamiliar cases well.

Technology♡ 0🔖 0