Large Language Models: Architectures, Pretraining, and Roadmaps

Chapters 1 & 2: A foundational guide defining the GPT path, transformer decoders, and the three-stage implementation blueprint

Download the entire book using the URL at the end of this article!

A large language model is a deep neural network trained on a very large text corpus so it can learn patterns in language and use those patterns to generate or respond to text. The phrase sounds broad because the capability is broad: the same model can complete a sentence, answer a question, rewrite a paragraph, or continue a conversation. What makes it a language model is not that it contains hard-coded grammar rules, but that it learns statistical structure from text itself. What makes it large is not just that the network has many parameters, but also that it has been exposed to a large volume of training data.

That distinction matters. A model with millions or billions of parameters can still be trained on a narrow dataset, and a small model can be trained on a huge corpus. In practice, modern LLMs are defined by both scale dimensions together. Parameter count gives the model capacity to store and transform patterns, while training data scale gives it the breadth of language experience needed to learn those patterns in the first place. The result is a model that does not merely memorize phrases; it learns reusable regularities across syntax, semantics, style, and task structure.

This is also where LLMs differ from older text systems built around manual rules or feature engineering. Traditional machine learning often depends on a human deciding which signals matter, such as specific keywords, counts, or other hand-crafted indicators. Deep learning replaces much of that manual design with learned representations. Instead of telling the model which textual cues are important, you provide data and let the network discover useful internal features during training. For language tasks, that shift is especially powerful because text is messy, contextual, and full of exceptions that are difficult to capture with fixed rules.

Where LLMs sit in the AI hierarchy

It is useful to place LLMs inside the broader field hierarchy. Artificial intelligence is the largest umbrella term here. It includes any approach aimed at building systems that perform tasks associated with intelligent behavior. Machine learning is a subset of AI focused on methods that learn from data rather than following only explicitly programmed rules. Deep learning is a subset of machine learning based on multilayer neural networks that learn representations through many stages of transformation. LLMs live at the intersection of deep learning and natural language processing, and they are one of the most visible examples of what deep learning can do at scale.

This hierarchy is more than taxonomy. It explains why LLMs emerged when they did. Earlier AI systems often relied on symbolic reasoning, expert rules, or carefully engineered statistical features. Those methods were useful, but they struggled to generalize across the enormous variability of human language. Deep learning offered a different route: learn the representation directly from data, then use that representation to solve downstream tasks. LLMs take that idea further than earlier neural language models by combining scale, architecture, and data to produce models that are useful across many language tasks without task-specific redesign.

Why next-word prediction matters

The basic training signal behind a language model is next-word prediction, often framed more precisely as predicting the next token in a sequence. During training, the model sees a prefix of text and learns to estimate what should come next. That simple objective turns ordinary text into supervision. No person needs to label every example with a class or answer key, because the structure is already present in the text itself. This is why language modeling is described as self-supervised learning: the input text provides its own training target.

The power of next-word prediction is easy to underestimate. To make good predictions, the model has to internalize a wide range of patterns, from short-range syntax to longer-range meaning and discourse flow. It must learn which words tend to follow which contexts, how tone changes across settings, how formal and informal phrasing differ, and how facts are usually expressed. The model is not trained to memorize isolated continuations; it is trained to build a general representation of language that supports many different continuations.

That same mechanism also explains why language modeling is such a strong foundation for later adaptation. A model trained to continue text has already absorbed a broad model of language, so it can be repurposed for tasks that require understanding, generation, or both. The next section of the chapter will build on this idea by separating the broad training phase from later task-specific adaptation.

Why these models are useful

The practical value of LLMs comes from flexibility. Once trained on sufficiently broad text, the model can often handle new prompts without having been explicitly trained on that exact task. That is the basis of zero-shot behavior, where a model is asked to perform a task with no examples beyond the prompt itself. Few-shot behavior goes a step further by providing a small number of examples in context so the model can infer the pattern to follow. These capabilities are not magic; they are consequences of large-scale learning from diverse text and the model’s ability to generalize from patterns it has already internalized.

This is why LLMs matter in practice. A single model can support many text-centric workflows, especially when the task can be expressed in natural language. The same underlying engine that predicts the next word can also generate drafts, rewrite content, summarize passages, or assist with conversational querying. The book’s later chapters will focus on how to build that kind of system, but the key point here is conceptual: an LLM is valuable because it learns a transferable language representation rather than a single narrow rule set.

A simple everyday example

Consider writing a short reply to a colleague who asks whether a meeting should move to Friday. A traditional system would need explicit logic for every phrasing variation and every possible context. An LLM, by contrast, can read the prompt, infer the intent, and continue with a response that fits the tone and the situation. It does this because it has learned from large amounts of text how requests, confirmations, and scheduling language are typically expressed. That ability to generalize from language patterns is the core reason LLMs have become so useful, and it is the foundation for everything that follows in the book.

Pretraining as the foundation

Pretraining is the stage where a language model first learns the broad patterns of text before it is asked to solve a narrower task. The core objective is next-word prediction. Given a sequence of tokens, the model learns to predict what token is likely to come next. That training signal is simple, but it is surprisingly rich because it can be extracted directly from raw text. No separate annotation step is required, and no task-specific label set has to exist in advance. The text supplies its own supervision, which is why this approach is called self-supervised learning.

This matters because pretraining is not trying to teach the model one skill at a time. It is building a general language capability from many examples of ordinary text. Over time, the model picks up syntax, phrasing, word relationships, and longer-range regularities that make language coherent. The result of this stage is a pretrained base model, sometimes called a foundation model, that can continue text in a broadly useful way even before it has been adapted to any specific use case.

The idea of a foundation model is central to the rest of the training lifecycle. Once a model has learned general language behavior from raw text, later training can focus on shaping that ability toward a narrower goal instead of recreating it from zero. That separation between general language learning and task-specific adaptation is what makes the pretraining-first workflow so effective.

Why scale and diversity matter

Pretraining only becomes powerful when it is done on a large and varied corpus. A small dataset can teach local patterns, but it will not expose the model to enough different writing styles, topics, and contexts to support broad reuse. Large corpora help the model see the same language mechanisms in many settings, which improves its ability to generalize. Diversity matters for the same reason. When the training text spans many domains and forms of expression, the model is more likely to learn reusable structure rather than memorize a narrow slice of language.

This is part of why pretrained models are expensive. Modern language models contain many parameters, and fitting those parameters to a large corpus takes substantial compute, time, and engineering effort. That cost is not a side issue; it is one of the reasons pretraining is so valuable. If a model already required a major investment to learn general language behavior, then reusing those learned weights is far more practical than repeating the same expensive process for every new application.

The payoff is that a pretrained model becomes a reusable starting point. It can support a wide range of downstream uses because it already encodes broad linguistic competence. The model is not specialized yet, but it has enough general knowledge to make later adaptation efficient. This is the main reason large-scale pretraining sits at the base of the workflow: it creates a model worth reusing.

Fine-tuning as task adaptation

Fine-tuning is the second stage of the lifecycle. Instead of learning language from scratch, the model begins with the pretrained base model and then continues training on data that reflects a narrower objective. That objective might be a domain-specific corpus, a set of labeled examples, or paired inputs and outputs that express the behavior you want. The point is not to relearn language, but to steer an already capable model toward a particular task or domain.

This distinction is easy to miss, but it is crucial. A model trained only on a small task dataset usually has to spend its capacity on both general language understanding and task adaptation at the same time. That is inefficient and often limiting. A pretrained model has already done the hard work of absorbing broad language structure, so fine-tuning can focus on specialization. In effect, the model shifts from broad language learning to narrow task learning.

That is why the two stages belong in this order. Pretraining comes first because it builds the reusable foundation. Fine-tuning comes later because it adapts that foundation to a concrete use case. Training from scratch on every downstream problem would be wasteful, especially when the model already spent substantial compute to learn general patterns. Pretraining makes the later step cheaper, faster, and more reliable.

This broad-first, specialized-second pattern is the organizing idea for the rest of the book. Pretraining gives the model its general language ability. Fine-tuning turns that ability into something practical for a specific application. Once you keep that distinction in mind, the rest of the pipeline becomes easier to understand and easier to reuse in later chapters.

Transformer architecture as the common starting point

Transformers are the architectural family that underlies most modern large language models. That statement matters because it separates the broad idea of an LLM from the specific design choices that make one model better suited for classification, another for translation, and another for open-ended text generation. In this book, the word transformer is the umbrella term, while GPT refers to one particular transformer design that we will build, train, and use throughout the later chapters.

At a high level, the transformer replaced earlier sequence-processing approaches by letting tokens interact directly with one another through self-attention. Instead of reading text strictly from left to right and compressing everything into a single hidden state, the model can decide which earlier tokens are most relevant for the current token representation. That one change is what makes transformers so effective at capturing context, long-distance dependencies, and subtle relationships inside text.

Encoder, decoder, and the role of self-attention

The original transformer organizes computation into two parts: an encoder that reads the input sequence and a decoder that produces the output sequence. The encoder builds contextual representations of the source text, and the decoder uses those representations to generate target text one step at a time. Self-attention sits inside both sides of that design and is the mechanism that lets each token incorporate information from other tokens in the sequence.

A useful way to think about self-attention is that it gives the model a learned routing system for context. When the model processes a word, it does not have to rely only on nearby words or a fixed memory window. It can place more weight on whichever earlier tokens are most useful for the current prediction. That flexibility is the reason transformers are so effective on text tasks where meaning depends on relationships across an entire passage rather than on local patterns alone.

Where BERT-like and GPT-like models diverge

Once the transformer family was established, two major directions emerged. One direction keeps the encoder and turns it into a strong representation learner. BERT-style models follow this path. They are trained with a masked-word prediction objective, which teaches the model to infer hidden tokens from their surrounding context. Because the encoder sees the whole input sequence at once, these models are especially effective when the goal is to understand text rather than to generate it. Classification, document tagging, and similar discriminative tasks fit this pattern well.

The other direction keeps the decoder and removes the encoder entirely. GPT-style models follow this path. Instead of reconstructing masked tokens from both sides of context, they are trained autoregressively to predict the next token from the tokens that came before it. This makes the model naturally suited to completion and generation, because the training objective matches the way the model will be used at inference time.

The difference is not only architectural, but behavioral. An encoder-focused model is built to produce a rich internal representation of an input. A decoder-only model is built to continue a sequence. That distinction becomes important as soon as you ask what the model should do with new text. If the task is sentiment classification or topic labeling, a BERT-like encoder is a natural fit. If the task is writing the next sentence, answering a prompt, or extending code, a GPT-like decoder is the better match.

A simple comparison makes the split concrete. Suppose the input is a product review. An encoder model reads the full review and produces a label such as positive or negative. A decoder-only model receives a prompt such as "Write a short summary of this review:" and then generates the summary token by token. The same source text can support both workflows, but the architecture shapes what the model is trained to do best.

Why GPT is the book’s implementation target

GPT is the model family this book will implement because it gives the cleanest path from transformer theory to a usable large language model. Its decoder-only structure is simpler than the original encoder-decoder layout, yet it is powerful enough to support the behaviors people now associate with LLMs: completion, rewriting, translation, summarization, and many forms of reasoning-by-prompting. That simplicity is practical as well as conceptual. Fewer moving parts make it easier to understand the model, train it from scratch, and inspect how each component contributes to the final result.

GPT is trained with self-supervised next-token prediction, which means the learning signal comes from the text itself rather than from manually labeled examples. This matters because it lets the model learn from enormous collections of ordinary text. Once trained on enough data and enough parameters, the model often acquires capabilities that were never explicitly programmed into it. It may respond sensibly to prompts it has not seen before, perform zero-shot tasks without examples, or improve further when given a few demonstrations in the prompt. Those behaviors are not magic; they are the result of scale, diverse pretraining data, and an objective that rewards learning general language structure rather than memorizing one narrow task.

That same scaling also gives rise to emergent behavior, meaning skills that appear only after the model reaches sufficient size and training breadth. In practice, this is one of the main reasons GPT-style models changed the field. A model trained only to predict the next token can end up handling far more than next-token completion when it is large enough and trained well enough. The exact boundary of what emerges is not fixed, but the pattern is clear enough to shape the modern LLM workflow.

For the rest of the book, GPT is the target model because it provides a direct path from first principles to a working language model. We will start with text data, build the token and embedding machinery it needs, implement attention and the transformer block, assemble the decoder-only network, and then train and adapt it for downstream tasks. Understanding why this architecture is the center of the book makes the later implementation chapters much easier to place: every component we build serves the same autoregressive generative model.

---

From roadmap to implementation milestones

The book’s three-stage implementation roadmap

A large language model is easiest to understand when it is built in the same order that the book will use to construct it. Rather than treating tokenization, attention, pretraining, and fine-tuning as disconnected subjects, the implementation is organized as a sequence of milestones. Each milestone prepares the ground for the next one, and the full system only makes sense once all three stages are in place.

The central blueprint throughout the book is a GPT-style decoder-only model. That choice matters because it gives the implementation a clear direction. We are not building a general survey of neural architectures or a family of unrelated language systems. We are building one specific kind of model, then following the path from raw text to a pretrained base model and finally to task-specific adaptation. The roadmap is deliberately narrow so that every chapter contributes to the same working system.

Stage 1: prepare the data and implement attention

The first stage establishes the infrastructure that makes the model trainable at all. Before a GPT-like model can predict the next token, text has to be turned into a form that a neural network can process. That means preparing text data, organizing it into token sequences, and setting up the inputs the model will eventually consume during training. This early work is not a side topic. It is the point where language becomes model input.

At the same time, the book introduces the attention mechanism, which is the core computational idea behind the transformer family. The reason attention appears so early is simple: without it, the rest of the model would have no meaningful way to relate one token to another across a sequence. Preprocessing and attention form the foundation of the entire implementation. Once they are understood and working, the later chapters can focus on assembling larger systems rather than inventing the basic machinery from scratch.

Stage 2: pretrain a GPT-like model and learn basic evaluation

The second stage turns the components into a trainable language model. Here the book shifts from preparation to learning, using the GPT-style decoder-only architecture as the training target. Pretraining is the phase that gives the model its general language competence. It is also the phase that turns the architecture into a foundation model, meaning a base model that can later be reused for other tasks instead of being built anew each time.

This stage also introduces evaluation in its simplest practical form. Once training begins, the question is no longer only whether the code runs, but whether the model is improving in a measurable way. The book therefore treats evaluation as part of the pretraining workflow rather than as an afterthought. That gives the reader an early habit that matters throughout the rest of the book: train, inspect, compare, and only then decide whether the model is moving in the right direction.

Pretraining is expensive in the real world, especially when the goal is a large model trained on massive data. For that reason, the book uses small-scale examples when it demonstrates training from scratch. Those examples are pedagogical, not industrial. They show the mechanics of the process and the role of each training step without pretending to reproduce the compute budgets of a full-scale production system. Where it helps the workflow, the book also shows how to load openly available pretrained weights, which makes it possible to study and adapt a capable base model without paying the full cost of pretraining every time.

Stage 3: adapt the pretrained model for downstream use

The third stage begins after a pretrained base model exists. At that point the model has learned broad language patterns, but it still needs adaptation before it becomes useful for a specific application. Fine-tuning provides that adaptation. In this book, fine-tuning is used to turn the pretrained GPT-style model into a system that can follow instructions and into a model that can perform text classification.

This stage is where the idea of reuse becomes concrete. A foundation model is valuable because it is not the final product. It is the starting point for downstream work. Fine-tuning lets the same pretrained backbone serve different goals without rebuilding the language model from the ground up. The model becomes narrower in purpose but more useful in practice, because its general linguistic knowledge is redirected toward a task the user actually cares about.

The downstream examples matter because they show two different kinds of adaptation. Instruction following emphasizes interactive behavior and response generation. Classification emphasizes structured prediction over a fixed label space. Together they demonstrate that the same pretrained core can support more than one kind of application, as long as the adaptation stage is designed with the task in mind.

Why the stages are ordered this way

The ordering is not arbitrary. Data preparation must come first because the model cannot learn from text it cannot consume. Attention must come early because it is the mechanism that gives the decoder-only transformer its power. Pretraining comes next because it is the step that produces a general-purpose base model. Fine-tuning comes last because it only becomes meaningful once that base model exists.

This dependency chain is the organizing principle of the book. Each stage solves a different problem, but the later stages depend on the earlier ones being in place and understood. By the time you reach the adaptation chapters, you will already have the vocabulary, the architecture, and the training picture needed to understand what is changing and why. That is the reason for the roadmap: it turns a large technical subject into a sequence of concrete milestones that build toward a working GPT-style system.

Chapter takeaways at a glance

This chapter established the vocabulary the rest of the book will use. An LLM is a transformer-based language model trained to predict text autoregressively, one token at a time. That simple formulation is the center of gravity for everything that follows, because it explains both why these models can generate coherent sequences and why they need large amounts of text to learn effectively.

What an LLM is and how transformer models fit

The most important structural idea is that LLMs are not a separate species of model; they are language models built on the transformer architecture. The transformer supplies the mechanism that makes large-scale sequence modeling practical, especially through self-attention. Self-attention lets the model use the surrounding context in a sequence while it is producing the next token, so generation is informed by the full available history rather than by a narrow local window.

That relationship matters because it connects the high-level promise of LLMs to the mechanics we will implement later. The model is not simply memorizing text. It is learning how to use context, representation, and probability to continue a sequence in a way that generalizes across many forms of language use.

GPT-style decoder-only models versus the broader transformer family

The transformer family is broader than the GPT-style models this book builds. The original transformer design included encoder and decoder components, which are useful in settings where one sequence must be read and another sequence produced. GPT-style models, by contrast, use only the decoder side of the architecture and are trained for left-to-right generation.

That distinction is more than terminology. A decoder-only model is aligned with the autoregressive objective that defines modern text generation systems. It predicts the next token using the tokens already seen, which makes it the natural architecture for a book focused on building a GPT-style model from scratch. Later chapters will work within this narrower design rather than treating the transformer as a generic all-purpose template.

From pretraining to foundation-model reuse

The chapter also introduced the two-phase lifecycle that gives an LLM its practical value. First comes pretraining on a large corpus of unlabeled text. This stage teaches the model general statistical structure, broad language patterns, and many regularities that are not tied to a single supervised task. After that, the same base model is adapted to a downstream task through fine-tuning or another form of task-specific specialization.

This is where the idea of a foundation model becomes useful. A pretrained model is not the end of the workflow; it is a reusable starting point. Once the base model has absorbed enough general structure, it can be specialized for narrower goals such as classification or instruction following. Large-scale pretraining can also produce capabilities that were never directly supervised, because breadth and scale let the model internalize patterns that transfer into many downstream uses. The important point is that later adaptation depends on the earlier foundation, since the base model’s broad pretraining is what makes it generally useful before task-specific supervision is applied.

Bridge to the next chapters

With these distinctions in place, the next chapters can move from concepts to construction. The book will start with data preparation, then build attention step by step, and from there assemble and train the GPT-style model itself.

Use the url below to download the entire book