Context Windows, Memory, and Limitations

The Desk Analogy

Imagine you have a desk where you can spread out papers while working. The desk is large but finite. You can lay out maybe 50 pages at once — reading them, cross-referencing, making notes. But the moment you need page 51, you have to remove an earlier page to make room. Whatever was on that removed page is gone from your immediate awareness.

This is essentially how a context window works. It's the total amount of text a model can "see" at once — your messages, its responses, any documents you've pasted in, and the system prompt. Everything the model considers when generating its next response must fit on this desk.

What Is a Context Window?

A context window is measured in tokens (roughly 0.75 words per token). Different models have different context window sizes:

• Early GPT models: ~4,000 tokens (about 3,000 words)
• GPT-4: 8,000–128,000 tokens depending on version
• Claude: up to 200,000 tokens (roughly 150,000 words — about the length of two novels)

Everything in your conversation — every message you've sent, every response the model has given — occupies space in the context window. As a conversation grows, it fills up.

Why Models "Forget"

If you've had a long conversation with an AI and noticed it seems to "forget" something you told it earlier, this is why. The earlier messages may have been pushed out of (or deprioritized within) the context window.

Important: models don't have persistent memory between conversations. Each new conversation starts from scratch. The model doesn't remember what you talked about yesterday, or even an hour ago in a different conversation. There's no personal memory bank.

Within a single conversation, the model can reference everything in the context window. But it doesn't "remember" in the human sense — it simply has access to the text. If you tell the model your name on message 1 and ask "What's my name?" on message 50, it can answer — but only because your original message is still within the context window, not because it formed a memory.

An AI model doesn't remember — it reads. Every time it generates a response, it re-reads the entire context window from scratch. There's no accumulating knowledge or growing understanding within a session — just a fixed window of text it can see.

Strategies for Working Within Context Limits

1. Front-Load Important Information

Put your most important context, instructions, and constraints at the beginning of the conversation. Information at the start of the context window tends to be attended to more reliably than information buried in the middle of a long conversation.

2. Summarize Rather Than Accumulate

If you're working on a long task, periodically ask the model to summarize the current state. Then start a new conversation with that summary as context. This is more token-efficient than continuing a sprawling conversation.

3. Paste Relevant Content Directly

Don't assume the model knows what's in a file or document. Paste the relevant text directly into the conversation. The model can only work with what's in its context window — not what's on your hard drive.

4. Be Aware of the Budget

If you're working with a 128K token context window and you've pasted in a 50,000-word document, you've used roughly 65,000 tokens just on the document. That leaves about 63,000 tokens for the rest of the conversation (your questions, the model's responses). Plan accordingly.

5. Break Large Tasks into Smaller Ones

Rather than asking the model to process an entire 200-page report at once, break it into sections. Process each section separately, then synthesize the results. This gives better results and avoids context window pressure.

Other Limitations

No Real-Time Information

Models are trained on data up to a specific date. They don't know what happened yesterday unless given that information. If you need current information, you need to provide it or use a model with web search capability.

No Learning from Conversations

Nothing you tell an AI model in a conversation changes its underlying model. It won't get "better" at your specific needs over time through conversation alone. Each conversation is independent.

Probabilistic, Not Deterministic

Ask the same question twice and you may get different responses. This isn't a bug — it's how sampling and temperature work. The model generates probabilistically, not by looking up fixed answers.

Key Takeaways

• A context window is the total text a model can consider at once — like a desk you can spread papers on.
• Models have no persistent memory between conversations. Each session starts fresh.
• Within a conversation, models don't "remember" — they re-read the entire context window each time.
• Front-load important info, summarize long conversations, and break large tasks into pieces.
• Models don't have access to real-time information and don't learn from individual conversations.