Tokens, Attention, and How Models Generate Text

Words Are Not What Models See

When you type a message to a language model, you see words. But the model doesn't process words the way you think. Before your text reaches the model, it gets broken into smaller pieces called tokens.

What Are Tokens?

A token is the basic unit a language model reads and generates. Tokens are often words, but not always — they can be word parts, characters, or punctuation. A model's vocabulary is typically 30,000–100,000 tokens.

Here's how a tokenizer might break down text:

• "The cat sat on the mat" → ["The", " cat", " sat", " on", " the", " mat"] — six tokens.
• "unbelievable" → ["un", "believ", "able"] — three tokens (split into meaningful parts).
• "GPT-4" → ["G", "PT", "-", "4"] — four tokens (technical terms get split).

Why Not Just Use Whole Words?

Using whole words would require millions of vocabulary entries. And the model would be stumped by any new word. Subword tokenization solves this: common words get their own tokens, uncommon words get broken into familiar pieces. "Unhappiness" becomes ["un", "happiness"] — even if the model has never seen "unhappiness" as one unit, it understands the parts.

It's like how you understand "ungooglable" despite never seeing it before. Your brain breaks it into "un-" (not) and "googlable" (able to be googled).

Why Tokens Matter to You

Context windows (how much text a model can consider) are measured in tokens. API pricing is per token. As a rule of thumb: one token ≈ three-quarters of a word in English.

The Attention Mechanism

Once text is tokenized, the model needs to understand how tokens relate. This is where attention comes in — arguably modern AI's most important innovation.

The Problem It Solves

Consider: "The dog that chased the cat across the yard was tired." "Tired" describes the dog, not the cat or yard — but the dog is nine words earlier. How does the model connect them?

Or: "The bank was steep" vs. "The bank was closed." "Bank" means different things depending on context.

Attention solves both by letting every token look at every other token and calculate relevance.

Attention as a Dinner Party

Imagine a dinner party with 20 people talking. You focus on the person speaking to you, peripherally notice the person next to them, and barely register the far side of the room. But if someone across the room says your name, your attention snaps to them.

Attention works similarly. For each token, it calculates an attention score for every other token — how much influence each should have. High relevance gets a high score; irrelevance gets a low score.

Multi-Head Attention

Models don't have just one attention pattern — they have many running in parallel, called attention heads. Each focuses on different relationships:

• One might track grammar (which noun a verb refers to).
• Another tracks meaning (semantically related words).
• Another tracks coreference ("she," "the doctor," and "Sarah" are the same person).

A large model might have 96+ attention heads, each learning its own way to connect tokens.

Attention is what lets a model understand context. Without it, "cold" would mean the same thing in "cold weather," "cold response," and "cold case." With attention, the model grasps the distinction.

How Text Generation Works

When you send a prompt, here's the pipeline:

1. Tokenize your text.
2. Process through dozens of layers, each refining understanding via attention.
3. Predict probabilities for every token in the vocabulary. "the" might get 12%, "a" gets 8%, "banana" gets 0.001%.
4. Select one token from those probabilities.
5. Repeat — add the selected token and run the whole model again for the next one.

Temperature: The Creativity Dial

After calculating probabilities, how does the model choose? This is where temperature comes in.

• Low temperature (0.1–0.3): Strongly favors high-probability tokens. Output is focused and predictable. Good for factual questions and coding.
• Medium temperature (0.5–0.8): Balances likely and unlikely tokens. Natural, varied output. Typical for conversation.
• High temperature (0.9–1.5): Gives more weight to unlikely tokens. Creative and surprising, but potentially incoherent.

Given "The sunset painted the sky in shades of ___":

• Low temperature → "orange" (most probable, every time)
• Medium → "gold" or "pink" (varied but sensible)
• High → "lavender" or "electric" (surprising, poetic, riskier)

When a model writes something creative — an unexpected metaphor, an elegant phrase — it's not inspiration. It's probability distributions and temperature settings selecting a less-obvious-but-fitting token from the model's vast pattern library.

Why Models Hallucinate

Understanding generation explains hallucination. The model always selects probable next tokens. If its training data contained "The capital of Australia is Sydney" (a common misconception), that pattern is encoded in its parameters. When asked, "Sydney" might get a high probability — even though Canberra is correct.

There's no internal fact-checker. The model generates text that looks like a correct answer. Usually it is. Sometimes it isn't.

Key Takeaways

• Tokens are what models actually process — often words but sometimes word fragments. One token ≈ 0.75 words.
• The attention mechanism lets every token assess its relationship to every other token, enabling context understanding.
• Text is generated one token at a time, with the full model running once per token.
• Temperature controls randomness — low for accuracy, high for creativity.
• Hallucination happens because models generate statistically probable text, not verified facts.