How an AI Model Gets Built
Pre-training → fine-tuning → RLHF, in order
① Building a model takes three stages: pre-training (creates language ability) → fine-tuning (shapes it to a purpose) → RLHF (tunes tone and manner). Cost is overwhelmingly concentrated in stage one.
② Pre-training solves exactly one kind of problem — "guess the next token." Solving it trillions of times produces grammar, general knowledge, and something resembling reasoning as byproducts.
③ Once the three stages finish, the parameters freeze. Whatever you tell it in a conversation holds only inside that conversation; nothing persists into the model.
① What "training" actually refers to
How LLMs work covered how a model produces an answer — splitting tokens, converting to coordinates, referencing each other via attention, sampling from a next-token distribution. That article focused on using a finished model (inference) and gave the building process a single paragraph.
This one fills that gap. And knowing the process resolves several follow-on questions automatically: why tens of thousands of GPUs get lashed together, why models differ in tone, and why "do you remember yesterday?" doesn't work.
| Stage | What it creates | Data | Relative cost |
|---|---|---|---|
| 1. Pre-training | Language ability itself | Raw text at internet scale | Overwhelming (months, hundreds of billions of won) |
| 2. Fine-tuning | The habit of following instructions | Tens to hundreds of thousands of Q&A pairs | A tiny fraction of stage 1 |
| 3. RLHF | Tone, manner, refusal boundaries | Human preference comparisons | Small, but human-labour intensive |
② Stage 1: Pre-training — there's only one problem
It's the most expensive and longest stage, yet the problem it solves is remarkably simple: "what token comes after this text?" That's all.
The procedure: take a chunk of text, hide what follows, have the model predict the next token, compare to the truth, and nudge the parameters in proportion to the error. Repeat trillions of times.
Ask what it takes to predict well. To continue "the capital of South Korea is" you need a fact. To continue "Cheolsu gave Younghee a book. The person who received the book is" you need to parse structure. To continue "3 + 5 =" you need something like arithmetic.
So grammar, general knowledge, and reasoning are byproducts, not objectives — capabilities that fell out of maximizing prediction accuracy. That distinction drives the limitations discussed later.
This is where GPUs become necessary
Hundreds of billions of parameters, each nudged slightly, trillions of times. And since the parameters don't fit on a single GPU, they're split across many, which then have to exchange intermediate results at every step.
So in a training cluster, communication speed between GPUs matters as much as raw compute. However fast one card is, it idles when data doesn't arrive from its neighbour. That's why HBM bandwidth and interconnect come up repeatedly on earnings calls — see our semiconductor sector deep dive.
Training is episodic and concentrated: one model occupies a huge cluster for months, then it's done. Inference is continuous and distributed: it recurs on every query for as long as the service is alive.
The two cycles therefore move differently. Whether an earnings call separates them tells you where that company sits — our Nvidia vs AMD vs Intel comparison traces how that split showed up in actual results.
What comes out of pre-training
A common misconception lives here. A model that has only been pre-trained — a base model — does not behave like a chatbot. Ask it a question and it may list more questions instead of answering. What it learned was "continue text", not "answer questions."
On the internet, what follows a question isn't only an answer. It's also other questions, ads, and forum signatures. The base model simply picks whichever is probable. Hence stage two.
③ Stage 2: Fine-tuning — teaching "questions get answers"
Fine-tuning trains the finished base model further on a much smaller dataset. The key ingredient is question–answer pairs: tens to hundreds of thousands of good answers written or reviewed by people.
What gets learned here isn't knowledge but form: questions are followed by answers, requested formats are honoured, "I don't know" is an acceptable output. A model past this stage is usually called instruction-tuned.
"Can't we fine-tune on our company documents to get an internal chatbot?" is a common instinct, and it tends not to work. Fine-tuning data is minuscule next to pre-training, so new facts don't implant reliably, and half-implanted knowledge becomes raw material for hallucination.
For internal documents, RAG — retrieving the document at question time and grounding on it — is the standard answer. It's more accurate to see fine-tuning as a tool for changing how the model behaves, not what it knows.
④ Stage 3: RLHF — encoding human preference
After stage two it answers questions. But among several valid answers, which is better has no answer key. The same content can be too verbose, too stiff, or needlessly assertive. You can't teach that from a table of correct answers.
So the method changes. RLHF (reinforcement learning from human feedback) runs roughly like this.
| Step | What happens |
|---|---|
| 1 | The model generates several answers to the same question |
| 2 | Humans rank which is better — comparisons, not scores |
| 3 | Those comparisons train a reward model that predicts human preference |
| 4 | The original model is adjusted to raise its reward-model score |
The fact that step 2 is comparison matters. "Score this answer out of 100" drifts between raters; "is A or B better" produces far more consistent judgments.
What emerges here is what we perceive as the model's personality: how polite it is, whether it asks a clarifying question or guesses, what it refuses. All decided here — which is also why models built on similar architectures and comparable data can feel so different to talk to.
People tend to rate confidently written answers more highly. Learn that preference and the model learns to write assertively even when its internal probabilities are low.
This is the technical basis for the claim in our hallucination guide that tone is not a reliability signal. A confident sentence is not the product of accuracy — it's the product of people having rewarded that style.
⑤ After the three stages — the parameters freeze
At deployment the parameters are fixed. However many people hold however many conversations afterwards, those numbers do not change.
That's why "remember what I told you yesterday?" fails. What you say holds only while it remains in that conversation's input. Close the window and it's gone. Even the "memory" features in recent services aren't learning — they write to separate storage and quietly re-insert it into the next conversation's input.
| Assumption | Reality |
|---|---|
| Chatting makes the model smarter | Parameters unchanged. Chat is just input |
| Facts I share will surface for other users | No learning occurs during inference |
| Memory features = learning | External storage, re-inserted into input |
| The model updates daily with current info | Knowledge stops at the cutoff; search fills the gap |
Parameters not changing during inference is a structural fact. But the conversations you type can be stored on a server and used as training data for a future model. That isn't architecture — it's each service's data policy.
If work documents are going in, checking whether training use can be disabled and how long conversations are retained comes first. That's why our AI tools comparison ranks it ahead of performance.
⑥ Summary — which stage decides what
· Doesn't know a fact → absent from pre-training data, or post-cutoff. Hard to patch by fine-tuning; retrieval (RAG) is the standard fix
· Ignores the requested format → fine-tuning's domain. Giving examples in the prompt recovers much of it
· Wrong tone / over-refuses → a disposition set during RLHF. The hardest part for a user to change
· States wrong answers confidently → a collaboration between pre-training (probabilistic selection) and RLHF (preference for assertive prose)
The three stages in one line: pre-training decides what it knows, fine-tuning decides what it does, RLHF decides how it speaks. When a "new model" appears in the news, reading it with those three separated makes it much easier to tell inflated announcements from real progress.
These same three stages are also where bias seeps in — from imbalances in the pre-training data to the standards of the people rating RLHF comparisons. How that happens at each stage is covered next in where AI bias actually comes from.
※ This article describes the general build process for publicly documented transformer-based language models as of August 2026. Individual commercial models differ in procedural detail. Companies mentioned are illustrative for the architecture and this is not investment advice.
※ This guide is provided for general educational purposes and simplifies technical details for readability.
New guides, when they land
We publish AI literacy guides twice a week. Subscribe and the next one comes to you — free, unsubscribe anytime.