What is a Language Model?

You Already Use a Language Model

Pick up your phone. Open a text message and start typing "I'll be there in." Your phone immediately suggests the next word: "five," "a," "ten," or "about." That suggestion is a language model at work — a system that predicts what word comes next based on the words that came before.

Your phone's autocomplete is a tiny, simple language model. ChatGPT, Claude, and Gemini are enormous, sophisticated language models. But they share the same fundamental principle: predict the next word.

That's it. That's the core idea. Everything else — the eloquent essays, the working code, the nuanced conversations — emerges from doing this one thing at extraordinary scale.

Language Has Patterns

The reason word prediction works at all is that language is deeply patterned. Consider these sentences:

• "The cat sat on the ___" → You'd predict "mat," "floor," "couch," or "chair."
• "She opened the door and ___" → "walked," "stepped," "saw," or "entered."
• "In conclusion, the evidence ___" → "suggests," "shows," "indicates," or "demonstrates."

You make these predictions effortlessly because you've absorbed the statistical patterns of English through a lifetime of reading and listening. Certain words follow other words with predictable frequency. Academic papers use different patterns than text messages. A recipe uses different patterns than a legal contract.

A language model does the same thing, but by processing billions of pages of text. It learns not just which words frequently follow which, but deep structural patterns: how sentences are constructed, how paragraphs flow, how arguments build, how stories unfold. It captures grammar, style, tone, factual associations, and logical relationships — all encoded as statistical patterns.

A language model doesn't understand language the way you do. It has discovered the statistical structure of language — the patterns that determine which words, phrases, and ideas tend to appear together — and it uses those patterns to generate text that follows the same structure.

From Phone Autocomplete to AI Assistant

Your phone's autocomplete predicts the next word based on the few words you just typed. It considers maybe 5-10 preceding words. It's useful but limited.

A large language model predicts the next word based on everything that came before — potentially tens of thousands of words. This includes your entire conversation, your instructions, and any context you've provided. It tracks relationships between words that are paragraphs apart.

This difference in scale creates an astonishing qualitative leap. When a model tracks enough context, word prediction starts to look like:

• Understanding questions: It predicts that after a question, the most likely text is a relevant answer.
• Following instructions: It predicts that after "Write a poem about autumn," the most likely continuation is... a poem about autumn.
• Reasoning: It predicts that after "If all dogs are mammals, and Rex is a dog, then Rex is..." the next word is "a mammal."
• Coding: It predicts that after a function signature and docstring, the most likely continuation is working code.

How Generation Works

When you send a message to an AI assistant, the model generates its response one word at a time (technically one token at a time — we'll cover tokens in a later lesson). The process:

1. The model reads your entire message.
2. It predicts the most likely first word of its response.
3. It adds that word to the sequence, then predicts the next word, considering everything including the word it just generated.
4. This repeats until the model generates a stopping signal or reaches a length limit.

Every word influences what comes next. The model doesn't plan its response in advance — it discovers its response as it generates, one word at a time. It's like a jazz musician improvising: each note shapes what comes after.

What "Large" Means

The "large" in Large Language Model refers to two things:

Large Training Data

Modern LLMs train on trillions of words — books, websites, academic papers, code, forums, and more. The entire English Wikipedia, which feels enormous, is a tiny fraction of a percent of a typical training dataset. This breadth is part of what gives these models their versatility.

Large Number of Parameters

Parameters are the internal numbers the model adjusts during training — its "knowledge storage." GPT-3 has 175 billion parameters. Newer models have even more. Each parameter captures some small facet of how language works. If each parameter were a grain of sand, 175 billion grains would fill roughly 70 dump trucks.

More parameters generally mean more nuanced understanding. It's like describing a face: 10 measurements give you a rough sketch, 10 million measurements give you a photorealistic portrait.

What Language Models Are Not

Understanding what language models aren't is just as important:

• They're not databases. Knowledge is distributed across billions of parameters, not stored in a table. This is why they can be subtly wrong about facts.
• They're not search engines. They don't retrieve information from the internet in real time (unless given that tool). They generate based on patterns learned during training.
• They're not conscious. Despite natural-feeling conversation, there's no inner experience or awareness.
• They're not infallible. They generate text that sounds like correct answers. Usually it is correct. Sometimes it isn't. This is called hallucination.

Key Takeaways

• A language model predicts the next word based on all preceding context — the same principle as phone autocomplete, but at enormous scale.
• Language has deep statistical patterns that models learn from billions of pages of text.
• Text is generated one word at a time, with each new word influenced by everything before it.
• "Large" refers to both the training data (trillions of words) and the parameters (billions of numbers encoding patterns).
• Language models are not databases, search engines, or conscious beings — they're sophisticated pattern predictors.