Tokens and the Context Window
Why pricing and limits are set the way they are
① A context window is not memory — it's a bowl. The entire conversation is re-sent every single turn, so each turn costs more as the conversation grows.
② Cost doesn't scale linearly with length. Attention scales with the square of the length — twice as long is four times the compute. That's the answer to "why are long documents so slow and expensive."
③ "One million tokens of context" describes what fits, not what the model can find. Holding text and reliably retrieving from the middle of it are separate problems.
① A token is neither a character nor a word
As covered in how LLMs work, a model can't handle characters directly — it splits text into tokens and numbers them. This article picks up the next question: why are pricing and limits denominated in this particular unit?
The reason is straightforward. A token is the smallest unit the model actually processes. Charging by character count would make the maths differ by language; charging by word count would drift away from the real work happening inside. Tokens are the most honest way to count what the model is doing.
Most AI services publish a tokenizer tool. Paste in text you actually work with and the intuition arrives quickly. The point isn't to memorize exact numbers — it's to measure, once, roughly how many tokens the kind of writing you do takes. With that sense, both cost and limits become predictable.
② The context window is a bowl, not a memory
This is where the most common misconception sits. It's natural to assume the model remembers earlier turns of a conversation. It doesn't.
The model starts from a blank slate every turn. What happens instead is that the service re-assembles the whole conversation and pushes it back in. Ask your tenth question, and everything from the first question through the ninth answer is submitted again.
| What goes into the context window | Common misconception |
|---|---|
| The system prompt (service-supplied instructions) | Invisible to you, but it occupies space |
| The entire conversation so far | Not "memory" — re-sent each time |
| Attached documents and images | Attached once, but included every turn after |
| Your current question | — |
| The answer the model will generate | Output draws on the same limit |
That last row matters more than it looks. The context window isn't input-only. Fill the bowl with input and there's no room left for the answer, so responses get truncated or come back oddly thin. If a summary of a long document feels lazy, this may be why.
The longer the thread, the more gets re-submitted each turn. Your twentieth question may be short, but processing it means processing the preceding nineteen turns again. Starting a fresh conversation when the topic changes helps both speed and cost. That's not a trick — it follows directly from the structure.
③ Why Korean costs more
The same sentence in Korean consumes more tokens than in English — commonly cited at around 1.5–2×. The cause isn't model capability; it's the composition of the data the tokenizer was trained on.
A tokenizer merges frequently-seen chunks into single tokens. If it was built on overwhelmingly English data, English words survive as whole tokens while Korean fragments into smaller pieces. Korean's rich particle and ending inflections compound the effect.
At volume, that gap is simply cost. Sending English in and asking for Korean out can be cheaper. But translation shaves nuance, so it's a poor fit where the texture of the prose matters. Which way works better is best settled by testing it yourself.
④ Why longer gets expensive fast — quadratic cost
This is the most practically useful part. Doubling the length feels like it should double the cost. It doesn't.
Attention computes how much each token should reference every other token. At 100 tokens that's 100×100 = 10,000 pairs; at 200 tokens, 200×200 = 40,000. Double the length, quadruple the compute. That's the quadratic (length²) cost of attention.
| Input length | Attention pairs | Multiple |
|---|---|---|
| 1,000 tokens | 1 million | baseline |
| 2,000 tokens | 4 million | 4× |
| 4,000 tokens | 16 million | 16× |
| 10,000 tokens | 100 million | 100× |
Real pricing doesn't trace this curve directly — various optimizations apply, and billing is usually a flat per-token rate. But why long inputs feel disproportionately slow, and why context limits can't simply be raised without end, both come from this structure.
More long-context work and more agents means tokens to process growing quadratically. That's why AI infrastructure spending keeps climbing on the inference side, not just training — the backdrop is in our semiconductor sector outlook, and the financing structure behind that spending is dissected in AI circular financing.
⑤ How to read a "1M token context" headline
Context limits grow with every model release. Reading the number alone, it's tempting to conclude you can now drop in an entire book. One distinction is needed first.
What fits and what can be found are different things. A large limit means your input won't be rejected. It is not a guarantee that everything inside it gets used evenly.
Across long inputs, models tend to hold the beginning and the end well while the middle goes soft. So effective retrieval matters more than the headline limit. Testing it is simple: plant one distinctive sentence in the middle of a long document and ask about it. That test, and what to do per document type, are covered in getting AI to read your documents.
In short, a context limit is necessary but not sufficient. A million tokens going in doesn't mean a million tokens understood.
⑥ Five practical ways to spend fewer tokens
Once the structure is clear, the savings follow from principle rather than trickery.
- New conversation when the topic changes — the biggest lever. It stops the whole prior thread from being re-sent every turn.
- Only the relevant part of a document — twenty pertinent pages beat a 200-page PDF on both speed and accuracy, and cut middle-of-document loss at the same time.
- Leave room for the output — fill the bowl with input and the answer gets truncated. Need a long answer? Send less.
- Summarize and restart long threads — carry forward just the conclusions so far, and you keep context while resetting length.
- Factor language into bulk work — over repeated runs, the Korean-versus-English gap accumulates.
A context window is not space the model remembers in — it's a bowl refilled from scratch every turn. Change that one idea and the rest follows: why long conversations slow down, why answers get cut off, and why a bigger limit isn't a cure-all all trace to the same mechanism.
※ Written as of August 2026, describing general principles. Per-token pricing and context limits vary by service and model and change often — check each provider's official documentation for current figures.
※ This guide is provided for general educational purposes and simplifies technical details for readability.
New guides, when they land
We publish AI literacy guides twice a week. Subscribe and the next one comes to you — free, unsubscribe anytime.