How Does ChatGPT Actually Answer?
No metaphors — just the computation, in order
① An LLM does not understand a sentence. It computes a probability distribution over the next token, one token at a time, feeding each result back in as input.
② The heart of that computation is attention — weighting how much each token should reference every other token. Its cost grows with the square of input length.
③ There is no separate mechanism that checks whether the output is correct. The model cannot distinguish a probable sentence from a true one — the structural root of hallucination.
① Why "it understands" is the wrong frame
Most explanations of AI open with a metaphor: it mimics the brain, it's like someone who read a lot of books. Metaphors feel helpful at first, then quietly create wrong expectations. Why does it fumble arithmetic? Why does it invent plausible-sounding papers? Why does a long document cost so much more? Metaphors can't answer any of these.
This article uses none. We follow the actual computation from the moment you hit enter to the moment characters appear on screen. No equations — but the structure is accurate.
② Step 1: Tokenization — turning characters into numbers
The model cannot handle characters. Only numbers. So the first thing it does is split your input into tokens and assign each a number.
Tokens are not words. Frequent chunks become a single token; rare ones get split. English "unbelievable" becomes un + believ + able.
| Input | Approx. tokens | Note |
|---|---|---|
| One English word | ~1 | Most common in training data |
| One Korean character | ~1–2 | Particles and endings fragment it |
| The number "1,234,567" | 4–6 | Often split by digit groups |
| One A4 page (Korean) | ~1,500–2,500 | Baseline for estimating cost |
API pricing and context limits are both measured in tokens. The same sentence in Korean routinely uses 1.5–2× the tokens of its English equivalent. For bulk processing, sending English and requesting Korean output can be meaningfully cheaper. Why pricing and limits are set this way is covered in tokens and the context window.
③ Step 2: Embeddings — turning numbers into coordinates
A token number carries no meaning on its own. Knowing that "apple" is #4823 tells you nothing about apples. So each token number is converted into an embedding — a list of numbers, effectively a coordinate in a space of thousands of dimensions.
What matters is the geometry of that space. Through training, tokens used in similar contexts end up near each other. "King" drifts toward "queen"; "Seoul" toward "Tokyo". Whatever "meaning" the model has is entirely this arrangement. There is no dictionary stored anywhere.
④ Step 3: Attention — the core of the architecture
This is the heart of the transformer. A word's meaning shifts with its neighbours. "Bank" in "river bank" and "bank account" are the same characters and entirely different things.
Attention is the operation where each token scans every other token in the sequence and computes weights — how much to reference each one. The token "bank" adjusts its own representation toward the geography region when it sees "river", toward finance when it sees "account".
This doesn't happen once. Dozens to hundreds of layers repeat it. Lower layers handle local patterns like grammar; higher layers handle the logical structure of the whole passage.
Every token references every token, so n tokens means n×n comparisons. Double the input and compute goes up 4×; ten times the input, 100×. That is why long documents are slow and expensive, and why context windows are finite.
⑤ Step 4: Next-token prediction — where generation actually happens
After the final layer, the model produces one thing: a probability distribution across the entire vocabulary. If the vocabulary holds 100,000 entries, you get 100,000 numbers, each the probability that this is the next token.
Given "The weather today is really", the output looks roughly like this:
| Candidate token | Probability |
|---|---|
| nice | 0.41 |
| hot | 0.22 |
| cold | 0.11 |
| clear | 0.08 |
| … the other 99,996 | 0.18 combined |
One gets chosen. Always taking first place makes text stiff and repetitive, so selection is usually randomized in proportion to probability. The knob controlling that randomness is temperature — lower for consistency, higher for variety.
Then the chosen token is appended to the input and the whole thing runs again from the start. That loop is why answers appear character by character. The model does not draft a full response in advance. At every moment it is choosing only the next fragment.
· Why different answers to the same question? — Sampling randomly from a distribution
· Why is arithmetic unreliable? — It isn't calculating; it's picking tokens that look like a calculation result
· Why do long answers drift? — Each generated token becomes input, so small early errors compound
· Why does "think step by step" improve accuracy? — Emitting intermediate steps as tokens puts them back into the input, giving later predictions more to work with. Reasoning models bake this in through training
⑥ When does learning happen? Pre-training vs inference
Everything above is inference — using a finished model. Building the model, pre-training, is an entirely separate process.
Pre-training runs the "guess the next token" task across vast text, trillions of times. Each miss nudges the parameters — the hundreds of billions of numbers used in the embeddings and attention computations. This consumes months and hundreds of billions of won worth of GPU time.
Critically, parameters never change during inference. What you tell the model in a conversation holds only within that conversation; the model itself learns nothing.
Two more stages follow pre-training — fine-tuning and RLHF — and that is where tone and refusal boundaries get set. The full build process is broken into three stages in How an AI Model Gets Built.
| Pre-training | Inference | |
|---|---|---|
| When | Once, before release | Every time you ask |
| Parameters | Continuously updated | Never change |
| Cost | Hundreds of billions of won | Cents per query |
| Hardware | Tens of thousands of GPUs + fast interconnect | A handful of GPUs |
Training and inference demand different hardware. Training bottlenecks on GPU-to-GPU transfer, making HBM bandwidth and interconnect critical; inference is about throughput and power efficiency. That's why chip earnings calls break out "training demand" and "inference demand" separately. See our semiconductor sector deep dive and Nvidia vs AMD vs Intel comparison.
⑦ So why does it hallucinate?
If you've followed this far, the answer is already in place. The model selects high-probability tokens, and nowhere in the pipeline is there a step that checks whether the result is true.
Ask for a paper that doesn't exist and the model produces a token sequence shaped like a paper title. Whether it exists is not a question the architecture asks. The more plausible the form, the higher the probability — so the most convincing falsehoods are the ones it generates best.
The apparent confidence is an illusion too. Tone follows the style of the training data and is unrelated to the internal probabilities. For how to handle this, see Why AI Hallucinates.
① Split input into numbered tokens → ② convert numbers to coordinates (embeddings) → ③ let tokens reference each other via attention and adjust those coordinates → ④ produce a next-token probability table and sample from it → ⑤ append the sampled token and return to ①. That's it. No comprehension, no verification, no intent anywhere in the loop.
This coordinate-space trick isn't limited to text. Convert images and audio into coordinates the same way and drop them into the same space, and attention keeps working without caring where each coordinate came from — that mechanism is covered next in multimodal AI.
※ This article describes the general architecture of publicly documented transformer-based language models as of July 2026. Individual commercial models differ in implementation details. Nothing here is investment advice regarding any company or product.
※ This guide is provided for general educational purposes and simplifies technical details for readability.
New guides, when they land
We publish AI literacy guides twice a week. Subscribe and the next one comes to you — free, unsubscribe anytime.