24 Large Language Models
Large Language Models (LLMs) have emerged as a defining technology in artificial intelligence. They write code, translate languages, analyze legal documents, and converse with fluency that feels human.
LLMs are built on the Transformer architecture (Vaswani et al. 2017), which made self-attention the sole mechanism for mixing information across a sequence. Attention itself is older (Bahdanau, Cho, and Bengio 2014); the Transformer’s contribution was dropping recurrence, so that every position attends to every other in parallel and a long-range dependency no longer has to survive a chain of recurrent updates. The mathematical foundations are detailed in Chapter 23; this chapter focuses on operational logic: from Transformer mechanics to emergent reasoning capabilities.
24.1 The LLM Lifecycle
Building and deploying an LLM involves decisions about what changes, which signal guides the change, and when the cost is paid. Alignment is an objective, such as following user intent while remaining honest, that can shape several training stages.
| Intervention | What changes | Required signal | When cost is paid |
|---|---|---|---|
| Pretraining | Model weights | Next tokens in a curated corpus | Training |
| Supervised adaptation or distillation | Weights or adapters | Demonstrations or teacher outputs | Training |
| Preference or verifier-based optimization | Policy weights | Ranked responses or checkable outcomes | Training |
| Prompting, search, retrieval, and tools | Context and computation around a fixed model | Instructions, evidence, or feedback | Each request |
| Serving optimizations | Numerical representation or execution | Calibration data and hardware measurements | Preparation and serving |
Data collection gathers and curates the training corpus. Pretraining learns next-token predictions (Section 24.3); instruction tuning adapts those predictions to requests (Section 24.4.1). Preference and verifier-based training use different signals (Section 24.4.3 and Section 24.4.5). Quantization changes numerical representation for serving (Section 24.7.2), while chain-of-thought prompting spends computation at inference (Section 24.4). Agentic applications that use model outputs to act are covered in Chapter 25.
To follow these distinctions, consider a fictional Alder shop assistant. Policy P1, effective 1 January 2025, permits EU customers to return unopened goods within 30 days of delivery. Policy P2, effective 1 October 2026, replaces P1 for orders placed from that date and permits returns within 14 days of delivery. Neither document specifies the rule for opened goods.
A customer whose order was placed and delivered on 2 October asks on 22 October whether an unopened item can be returned. An assistant repeating P1 says yes; applying P2 gives no. For opened goods, the evidence is insufficient. We will revisit this example to distinguish learning a response format, supplying current evidence, checking an answer, and deciding when to abstain. These are stipulated policies and illustrative answers, not measured model results.
Before changing the assistant, reserve unseen questions about both policy versions, boundary dates, and opened goods. Judge whether each answer applies the effective policy, supports its claims, and acknowledges missing information. This gives each intervention the same decision to improve (Section 24.6).
24.2 Building Intuition: Character-Level Text Generation
Character-level text generation strips the problem to its essentials. LLMs operate on tokens (typically subwords), but examining character-level patterns reveals the core insight behind statistical language modeling.
Now let’s count letter frequencies in the text and plot the 26 most frequent letters.
If we try to generate the text one letter at a time, sampling each letter in proportion to its frequency (letters and counts hold the 26 letters of the plot and their counts), we get gibberish:
import numpy as np
# Generate text one letter at a time by sampling from the letter frequencies
counts = np.array(counts)/sum(counts) # Normalize counts to probabilities
import random
random.seed(42) # Fix the seed for reproducibility
gentext = random.choices(letters, weights=counts, k=20)
print("Generated letters:", ''.join(gentext))
## Generated letters: reiidcpeneioeacsirmeWhat if we use bi-grams, i.e., pairs of letters? Counting the frequencies of each pair gives us a sense of how often each combination appears.
The Cat article makes letter and bigram counts easy to inspect. To examine longer contexts, we switch to Tolstoy’s larger novel “War and Peace,” estimating 2-grams, 3-grams, 4-grams, and 5-grams and generating the following continuations:
2-gram: ton w mer. y the ly im, in peerthayice waig trr. w tume shanite tem.
3-gram: the ovna gotionviculy on his sly. shoutessixeemy, he thed ashe
4-gram: the with ger frence of duke in me, but of little. progomind some later
5-gram: the replace, and of the did natasha's attacket, and aside. he comparte,We used the nltk package (Bird, Klein, and Loper 2009) to estimate the letter frequencies from Tolstoy’s novel and generate text based on those models. The results show that even with simple letter-based models, we can generate text that resembles natural language, albeit with nonsensical phrases. As the n-gram order increases, the generated text becomes more coherent: the 5-gram output even captures character names like “Natasha.” More context can improve prediction when there are enough data to estimate the conditional probabilities.
LLMs have much larger context windows: Gemini 3 Pro accepts up to 1 million tokens, approximately the size of Tolstoy’s “War and Peace.” Counting longer contexts soon encounters sparsity. There are \(26^{10}\) (over 141 trillion) possible 10-letter combinations. With a vocabulary of 40,000 words, there are 1.6 billion possible bigrams, 64 trillion trigrams, and 2.56 quintillion four-word combinations (\(40{,}000^4 \approx 2.56 \times 10^{18}\)). For a typical 20-word question to an AI chat, the number of possible word sequences exceeds the number of particles in the universe. We need not store every combination, but most long contexts have few or no observed continuations. A neural model replaces a separate count table for each context with shared parameters that map token representations to next-token logits. Training on one context can therefore improve predictions for related, unseen contexts.
Attention mixes information from the available token positions into context-dependent representations (see Chapter 23). In autoregressive Transformers, a causal mask prevents a position from attending to future tokens during training. The following demonstration inspects the distribution produced by an already trained model.
24.3 Pretraining and Generation: One Token at a Time
The most visible capability of LLMs is text generation, producing coherent, contextually relevant continuations from a given prompt. This is autoregressive modeling: an LLM is a probability distribution over sequences of text, trained to predict the next token given a preceding context. A token serves as the atomic unit of processing; depending on the tokenization schema, it may represent a full word, a subword unit (like “-ing”), or a single character. Given the prompt \(Q\) and a sequence of tokens \(x_1, x_2, \ldots, x_t\), the model samples the next token \(x_{t+1}\) from the conditional distribution \[ x_{t+1} \sim p_\theta(x_{t+1} \mid Q, x_1, x_2, \ldots, x_t). \] The conditional variables \(Q, x_1, x_2, \ldots, x_t\) are called the context. The prompt may include a question and attached documents. With weights \(\theta\), the chain rule gives the probability of a continuation:
\[ p_\theta(x_{1:T} \mid Q) = \prod_{t=1}^{T} p_\theta(x_t \mid Q, x_{<t}). \]
Pretraining minimizes negative log-likelihood, equivalently next-token cross-entropy, over a corpus \(\mathcal D\):
\[ \mathcal L(\theta) = -\sum_{x_{1:T}\in\mathcal D}\sum_{t=1}^{T}\log p_\theta(x_t \mid x_{<t}). \]
Each observed prefix supplies a prediction target, its actual next token. Shared neural parameters let the model generalize beyond literal prefix matches. At inference the weights are fixed: conditioning on a new policy document changes the distribution over tokens, not a Bayesian posterior over the weights. A token’s probability is also not the probability that the sentence containing it is true. Whether confidence predicts correctness is a calibration question to evaluate on held-out answers.
To inspect generation, we use the fixed, instruction-tuned SmolLM2 checkpoint, not a pretraining run. AutoTokenizer.from_pretrained and AutoModelForCausalLM.from_pretrained from transformers load HuggingFaceTB/SmolLM2-1.7B-Instruct into tokenizer and model, downloading the weights once and keeping them in half precision.
Consider the text The best thing about AI is its ability to. The model computes next-token logits from the context, rather than searching for literal matches in its training corpus. The helper get_next_word_suggestions, defined in code/llm_chapter.py, runs one forward pass and returns the top_k most probable next tokens with their softmax probabilities.
# Get next word suggestions for a given text
initial_text = "The best thing about AI is its ability to"
suggestions = get_next_word_suggestions(initial_text, model, tokenizer, top_k=5)
print(f"Next word suggestions for '{initial_text}':")
for i, (word, prob) in enumerate(suggestions):
print(f" {i+1}. '{word}' (prob: {prob:.3f})")## Next word suggestions for 'The best thing about AI is its ability to':
## 1. ' learn' (prob: 0.277)
## 2. ' help' (prob: 0.054)
## 3. ' augment' (prob: 0.045)
## 4. ' analyze' (prob: 0.039)
## 5. ' process' (prob: 0.034)The probabilities are computed over the full vocabulary, so they do not sum to one across the five shown: the remaining mass is spread over the other 49,000 tokens. Figure 24.1 plots the top ten on a log scale.
Past the top word ‘learn’, log-probability falls almost linearly in rank, so probability itself decays roughly geometrically, by about 10% per rank. Zipf’s law, observed by George Kingsley Zipf in the 1930s, describes a different decay: the frequency of a word is inversely proportional to its rank, a power law, which would be a straight line on log-log rather than log-linear axes. What the two share is concentration. A handful of continuations carry most of the mass, and the remainder is spread thinly over tens of thousands of tokens, which is what makes the choice of sampling rule matter.
The most probable continuation is not always the desired one. “The answer is yes” and “The answer is no” are both grammatical; the appropriate continuation depends on the question and evidence. In the Alder example, a fluent continuation based on P1 is wrong for a P2 order. Sampling alternatives can diversify answers, but it does not supply the missing policy.
Sampling can yield different outputs from the same prompt. Temperature \(T\) controls the spread of the distribution, as in the Boltzmann distribution in statistical physics. Given logits \(z_i\), token \(i\) has sampling probability
\[ q_i = \frac{\exp(z_i/T)}{\sum_j \exp(z_j/T)}, \qquad T>0. \tag{24.1}\]
Higher temperatures flatten the distribution. Greedy decoding selects a token with the largest logit; the expression “temperature zero” conventionally refers to this deterministic choice, rather than division by zero in Equation 24.1.
In code, one step of generation takes either the most probable token or a draw from the softmax of the logits divided by the temperature:
if sample:
probs = torch.softmax(next_token_logits.float() / temperature, dim=-1)
next_token_id = torch.multinomial(probs, num_samples=1).item()
else:
next_token_id = torch.argmax(next_token_logits).item()The following example starts from the same prompt (The best thing about AI is its ability to) and calls generate_text_step_by_step, also defined in code/llm_chapter.py, which appends one token per forward pass until it has added num_steps tokens or produced the end-of-text token, choosing the most probable token when sample=False, as here, and drawing from the softmax at the given temperature when sample=True:
# Generate text step by step
generated_text = generate_text_step_by_step(initial_text, model, tokenizer,
num_steps=10, temperature=1.0, sample=False, print_progress=False)
print("Generated text:")
print(textwrap.fill(generated_text, width=60))## Generated text:
## The best thing about AI is its ability to learn and adapt.
## It can analyze vast amounts ofIn this example, we always select the most probable next token, which leads to a coherent but somewhat predictable continuation. The model generates text by repeatedly applying this process, building on the context provided by the previous tokens.
Now we generate a longer continuation, sampling at temperature 0.8. This is an illustrative setting, not a task-independent recommendation.
# Fix the seed for reproducibility
torch.manual_seed(8)
generated_text = generate_text_step_by_step(initial_text, model, tokenizer,
num_steps=60, temperature=0.8, sample=True, print_progress=False)
print("Generated text:")
print(textwrap.fill(generated_text, width=60))## Generated text:
## The best thing about AI is its ability to learn from
## humans," said the AI.The continuation stays on topic and stops well short of the 60 requested steps: sampling reached the end-of-text token, and the loop halted there. Now let’s try a higher temperature of 1.2.
torch.manual_seed(8)
generated_text = generate_text_step_by_step(initial_text, model, tokenizer,
num_steps=60, temperature=1.2, sample=True, print_progress=False)
print("Generated text:")
print(textwrap.fill(generated_text, width=60))## Generated text:
## The best thing about AI is its ability to rethink our
## understanding of what human intelligence might mean,' says
## Dr Crowell. 'AI's reliance on sense perception to learn
## leads to interactions that are bubbling, fluid, and a little
## disconcerting, in ways that are provocative and familiar.'
## Visit andrewhaddyousay.columbThe flatter distribution keeps the model going far longer and the prose stays grammatical, but it wanders into invented specifics: an attributed quotation from a “Dr Crowell” and a web address. The output supplies no evidence for either. Lower temperature concentrates sampling on what the model considers most likely; a likely answer can still be false, and greedy decoding can repeat it consistently. These two continuations illustrate a change in sampling.
24.4 Learning and Inference
The same next-token model can be changed through training or used inside a more elaborate inference procedure. The distinction is the signal: demonstrations specify responses, preferences rank responses, and verifiers check outcomes. Prompting and search then use the resulting policy without updating its weights.
Fine-tuning
Fine-tuning adapts a pre-trained model to specific tasks or domains by continuing training on specialized data. A base model may continue the policy text instead of answering the customer; instruction tuning trains the response behavior. For Alder, supervised fine-tuning (SFT) could teach a response format containing an answer, a policy identifier, and an evidence quotation, including examples that abstain when information is missing. It need not memorize each new policy: current documents can be supplied at inference. How many demonstrations suffice depends on the starting model, task, and coverage of the data.
Full fine-tuning updates all model parameters, resulting in a brand-new version of the model optimized for the specific task. Full fine-tuning requires significant compute resources and memory, and risks catastrophic forgetting, where the model loses its general capabilities as it specializes for the new task.
Parameter-efficient fine-tuning (PEFT) methods modify a small subset of parameters while freezing most of the model. LoRA (Low-Rank Adaptation) learns a low-rank additive correction to each targeted weight matrix, reducing gradient and optimizer-state memory. Frozen base weights preserve the original checkpoint, but the combined model can still lose capabilities. At inference the correction can be folded into the weights without changing the architecture. LoRA decomposes the update into low-rank factors: \[ W' = W + BA \] where \(B \in \mathbb{R}^{d \times r}\) and \(A \in \mathbb{R}^{r \times k}\) with rank \(r \ll \min(d,k)\). Ranks such as \(r=8\) to \(64\) illustrate the reduction in trainable parameters; the rank and resulting task performance must be selected and evaluated.
Instruction fine-tuning trains the model with examples explicitly showing how it should respond to different queries. The labeled data consist of input-output pairs where inputs convey the desired behavior and outputs represent the expected responses. An example instruction is solve this math word problem step by step, with input Sarah has 12 apples and gives away 5. How many does she have left? and the target answer:
Step 1: Start with 12 apples.
Step 2: Subtract 5 apples given away.
Step 3: 12 - 5 = 7 apples remaining. Sequential fine-tuning gradually adapts a model to increasingly specialized tasks. For example, a general AI model could first be fine-tuned for medical terminology, then refined further for pediatric cardiology. This progressive specialization allows the model to build upon previously learned knowledge. Each stage still needs evaluation of new and retained capabilities.
Multi-task learning trains the model on datasets containing instructions for various tasks over multiple training cycles. Mixing earlier tasks with new ones can reduce forgetting, but does not guarantee retention.
Instruction tuning usually applies next-token cross-entropy to answer tokens, conditioned on the supplied instruction and input. It trains the model to imitate demonstrations (Chung et al. 2022). Flan-PaLM 540B improved over PaLM 540B across MMLU, BBH, TyDiQA, and MGSM, by an average of 9.4%, illustrating generalization across a task mixture. Domain-specific SFT can have narrower benefits: in ICD-10 coding, exact-code accuracy rose from less than 1% to approximately 97% on standard coding and 69% on real clinical notes (Hou et al. 2025). The gap matters: adapting vocabulary and format does not establish reliability on unfamiliar clinical text. Fine-tuning can improve adherence to requirements; external checks and task evaluation establish whether those requirements are met.
Model Distillation: Knowledge Transfer
Distillation addresses a deployment question: can a less expensive model reproduce a teacher’s useful behavior on the target task? For Alder, a teacher might generate candidate policy answers that are checked before being used to train a smaller assistant.
Model Distillation transfers knowledge from a large, complex “teacher” model to a smaller, more efficient “student” model. A smaller student can reduce memory and serving cost, but speed and accuracy depend on architecture, hardware, and task. Teacher errors can also be transferred.
Model distillation, formalized by Hinton, Vinyals, and Dean (2015) in their paper “Distilling the Knowledge in a Neural Network,” is based on a Teacher-Student architecture. The teacher’s probability distribution contains information beyond its final answer.
For an image labeled “Dog,” hard-label cross-entropy rewards probability on that class. A teacher distribution such as Dog: 0.90, Cat: 0.09, Car: 0.0001 also conveys relative similarity to other classes. These partial probabilities, sometimes called dark knowledge, are absent from the hard label.
Distillation trains a smaller student model to mimic these soft targets (the probability distributions) produced by the teacher. By doing so, the student learns how the teacher generalizes, not just what the teacher predicts. The student can also inherit the teacher’s mistakes.
Different approaches transfer different information:
Response-based distillation focuses on having the student mimic the final output layer of the teacher model. Feature-based distillation leverages the internal representations or features learned by the teacher in its intermediate layers. Relation-based distillation captures and transfers the relationship knowledge between data samples and layers within the neural network.
Distillation uses the same temperature-softmax transformation as Equation 24.1. Here a higher temperature exposes small teacher probabilities for learning, rather than diversifying generated text.
The student objective can combine a soft-target loss and a hard-label loss:
\[ L = \alpha L_{\mathrm{soft}} + (1-\alpha)L_{\mathrm{hard}}, \qquad L_{\mathrm{soft}}=T^2\operatorname{KL}(q^{\mathrm{teacher},T}\|q^{\mathrm{student},T}). \]
Here \(L_{\mathrm{hard}}\) is cross-entropy at \(T=1\), and \(\alpha\) balances teacher imitation against labeled outcomes. The conventional \(T^2\) factor compensates for the temperature’s effect on gradient scale.
The following diagram illustrates the distillation pipeline, where the student learns from both the dataset and the teacher’s soft outputs.
Offline distillation refers to the traditional approach where the teacher model is trained first, then the student model is trained separately using the soft labels generated by the teacher. Online distillation is used when a large pre-trained teacher is not available for a given task, or when the teacher is so large that storage or processing capacity is insufficient. The teacher and student are trained simultaneously, with the student learning from the teacher dynamically during training. Self-distillation is a variant where a single model acts as both teacher and student: knowledge is transferred from deeper layers to shallower layers of the same network, which can improve performance and regularization without a separate teacher. Because the shallow layers are trained to reproduce the deep layers’ outputs, the network can also be truncated at an intermediate layer at inference, trading accuracy for depth.
Sanh et al. (2019) demonstrated distillation’s practical impact: DistilBERT retained approximately 97% of BERT’s performance while reducing parameters by 40% and increasing inference speed by 60%.
Teacher-generated rationales can also become supervised training data (Hsieh et al. 2023). In production, a large model (for example 200B+ parameters) can generate synthetic data or soft targets that fine-tune a smaller, cheaper model (for example 8B) for a downstream task. Distillation can support compact models for mobile or edge devices, including MobileBERT, and e-commerce and recommendation platforms have applied privileged feature distillation to improve click-through and conversion rates while managing inference costs. Distillation is most effective after a model has been fully pre-trained or fine-tuned on a specific task; common cases are production deployment, edge and mobile applications, budget-conscious inference at scale, and replacing an overly large general-purpose model with a compact, focused one.
Alignment
Alignment is the process of training models to behave according to human values and intentions: to be helpful, harmless, and honest. A model that excels at next-token prediction may still produce harmful content, follow malicious instructions, or optimize for metrics that diverge from what users actually want. Supervised demonstrations, preferences, and AI feedback can all contribute to this objective.
The schematic illustrates neutrality on a contested religious question. The aligned response remains neutral, acknowledges diverse perspectives, and offers to be helpful without asserting one belief system over others. Neutrality is one target alongside honesty, robustness, and following user intent; the illustration does not establish how a particular checkpoint behaves.
Reinforcement Learning from Human Feedback (RLHF)
A common RLHF pipeline involves three stages. First, supervised fine-tuning starts with a pre-trained model and fine-tunes it on high-quality demonstrations of desired behavior. Second, reward model training generates multiple responses to prompts and has human annotators rank them; a separate model learns to predict these preferences, creating a reward model that scores any response. Third, policy optimization uses reinforcement learning, often Proximal Policy Optimization (PPO), to update the language model to maximize the reward model’s scores while staying close to the original model to limit drift.
This loop optimizes the model to produce outputs humans prefer rather than those that merely score well on next-token likelihood. Preference can reward clarity or agreement without detecting factual errors. In the original InstructGPT study by Ouyang et al. (2022), annotators preferred answers from the 175B RLHF-tuned model over the same-size GPT-3 baseline 85% of the time. Even the 1.3B RLHF model was preferred to that baseline. These are preference results under the study’s protocol.
Direct Preference Optimization (DPO)
While RLHF produces strong results, it is complex and computationally expensive, requiring training a separate reward model and running reinforcement learning with careful hyperparameter tuning. Direct Preference Optimization (DPO) (Rafailov et al. 2023) instead trains directly on preference pairs.
Under the KL-regularized reward objective, the optimal policy has a closed form. Substituting its implied reward into a pairwise preference model gives a classification-style loss for one preference pair:
\[ \mathcal{L}_{\text{DPO}} = -\log \sigma\left(\beta \log \frac{\pi_\theta(y_w \mid x)}{\pi_{\text{ref}}(y_w \mid x)} - \beta \log \frac{\pi_\theta(y_l \mid x)}{\pi_{\text{ref}}(y_l \mid x)}\right) \]
where \(y_w\) is preferred to \(y_l\), \(\pi_\theta\) is the policy being trained, \(\pi_{\mathrm{ref}}\) is a fixed reference policy, \(\sigma\) is the logistic function, and \(\beta>0\) controls regularization. For the derivation, fix a prompt \(x\) and maximize \(\E[y\sim\pi(\cdot\mid x)]{r(x,y)}-\beta\operatorname{KL}(\pi\|\pi_{\mathrm{ref}})\). The solution is \(\pi^*(y\mid x)\propto\pi_{\mathrm{ref}}(y\mid x)\exp(r(x,y)/\beta)\). Rearranging gives \(r(x,y)=\beta\log(\pi^*(y\mid x)/\pi_{\mathrm{ref}}(y\mid x))+c(x)\). In the Bradley-Terry model, \(P(y_w\succ y_l\mid x)=\sigma(r(x,y_w)-r(x,y_l))\). The prompt-dependent constant cancels, yielding the loss above. Training averages it over observed preference pairs.
DPO requires (prompt, preferred response, dispreferred response) triples and supervised learning infrastructure, without a separate reward model or an RL sampling loop. The original study reports competitive preference results (Rafailov et al. 2023); outcomes depend on preference quality, the reference policy, and the evaluated task. DPO has been adopted for open-weight models such as Llama 3 Instruct and Zephyr-7B.
Constitutional AI
Constitutional AI (CAI) (Bai et al. 2022) takes a different approach: rather than relying solely on human feedback for each decision, it encodes explicit principles, a “constitution,” that guide the model’s behavior. The model learns to critique and revise its own outputs according to these principles.
The CAI process works in two phases. In the first phase, Supervised Learning from AI Feedback (SL-CAI), the model generates responses including potentially harmful ones, then critiques its own response according to constitutional principles, revises based on that critique, and finally the system fine-tunes on these revised responses. In the second phase, Reinforcement Learning from AI Feedback (RL-CAI), the system generates response pairs, has the model evaluate which better follows the constitution, trains a reward model on these AI-generated preferences, and uses RLHF with this reward model.
Constitutional principles might include directives such as choosing the response least likely to harm individuals or society, supporting human autonomy and freedom, or prioritizing honesty and truthfulness. Anthropic’s Claude models use Constitutional AI as a core component of their alignment strategy. Its appeal lies in scalability and transparency: rather than labeling thousands of examples, practitioners define principles that govern model behavior, and when behavior goes wrong the constitution can be updated. A written principle is not an enforcement mechanism. Updating it changes the feedback specification; changing the trained model still requires adaptation and evaluation.
Safety Alignment
Beyond preference alignment, safety alignment specifically addresses preventing harmful outputs through several complementary techniques.
Red teaming involves adversarial testing where humans or other AI systems deliberately try to elicit harmful, biased, or dangerous outputs. Red teams probe for jailbreaks that bypass safety filters, harmful content generation, privacy violations and data extraction, deceptive behaviors, and bias amplification across demographic groups. Automated red teaming uses other models to generate attacks, but still requires checking whether they cause harmful outcomes.
Adversarial training takes discovered attacks and uses them to harden the model. Responses to adversarial prompts are labeled, and the model is fine-tuned to refuse appropriately.
Guardrails and filters provide runtime protection. Input classifiers detect potentially harmful prompts; output classifiers screen responses before delivery. These serve as defense-in-depth layers complementing training-time alignment. They can miss attacks or reject legitimate requests.
Refusal training teaches when to decline harmful requests while remaining helpful for legitimate queries. Both missed harmful requests and unnecessary refusals need evaluation.
Reward Hacking and the Alignment Tax
Reward hacking (also called reward gaming, after Goodhart’s law) occurs when optimization exploits weaknesses in the reward signal. Examples include sycophancy (agreeing with a false user claim, because agreement tends to earn higher preference ratings), verbosity bias (longer responses, because annotators often equate length with quality), format gaming, and satisfying a literal criterion while missing its intent. Diverse annotators, reward-model ensembles, reference-policy penalties, and explicit principles can reduce particular failures; none replaces independent evaluation.
The alignment tax is a possible loss of useful capability after alignment. Overbroad refusals can obstruct legitimate security research or policy analysis. The size and direction of the trade-off depend on the training method, refusal boundary, and task.
Capability-specific constraints and principle-based feedback seek to preserve legitimate assistance while limiting harmful behavior. In Alder’s assistant, tone preference should be evaluated separately from applying the correct policy and avoiding invented exceptions.
Data Quality and Quantity
Data quality matters at every stage, but a small adaptation dataset is not the model’s total training experience. The starting checkpoint and teacher already contain information learned from much larger corpora.
The s1 study (Muennighoff et al. 2025) fine-tuned Qwen2.5-32B-Instruct on 1,000 high-quality reasoning examples. Their data collection process was methodical: they started with 59,029 questions from 16 diverse sources (including many Olympiad problems), generated reasoning traces using Google Gemini Flash Thinking API through distillation, then applied rigorous filtering. Problems were first filtered by quality (removing poor formatting), then by difficulty: a problem was deemed difficult if neither Qwen2.5-7B-Instruct nor Qwen2.5-32B-Instruct could solve it, and the reasoning length was substantial. Finally, 1,000 problems were sampled strategically across various topics. Its results also use budget forcing at inference, so they do not isolate data curation from test-time computation.
LIMO (Ye et al. 2025) also starts from Qwen2.5-32B-Instruct. With 817 curated samples, it reports 63.3% on AIME24 and 95.6% on MATH500. These are results for additional adaptation of an already pretrained and instruction-tuned model, not learning general reasoning from 817 examples in total.
LIMO emphasizes structured organization, allocating tokens to individual thoughts according to importance and complexity; cognitive scaffolding, bridging concepts step by step; and rigorous verification of intermediate results and assumptions. The practical question is the marginal value of curated examples at fixed training and inference budgets.
The rStar-Math research (Guan et al. 2025) generates training traces by prompting an LLM to produce solutions as Python code with text as code comments. Execution checks calculations and supplies feedback, but running without an error does not prove that the code represents the problem correctly.
Figure 24.3 shows how code execution can supply intermediate checks. It connects imitation of solution traces to the distinct training signal considered next: whether a generated outcome passes a verifier.
Learning from Verifiable Outcomes
Return to Sarah’s \(12-5=7\) problem. SFT imitates the supplied worked solution. Preference learning can rank that response above a fluent but incorrect answer. Reinforcement learning from verifiable rewards instead samples solutions and rewards a checkable outcome, for example \(r=1\) when the parsed answer equals 7 and \(r=0\) otherwise. This uses the policy-optimization framework of Chapter 9 with a different source of reward.
Math answer checkers and executable coding tests can provide repeatable feedback without a person ranking every response. DeepSeek-R1 studies this approach. It uses Group Relative Policy Optimization (GRPO), introduced in DeepSeekMath, which compares rewards within a group of sampled answers to the same question, increasing the relative probability of above-average outcomes while constraining policy changes (DeepSeek-R1). Unlike SFT, the successful trace need not be supplied in advance.
The verifier defines the incentive. A program can pass incomplete tests while failing untested cases; a correct final number does not validate every written step. Such gaps permit reward exploitation. Success on checkable math or code outcomes therefore does not establish reliability on an open-ended policy question whose evidence is incomplete.
Inference-time Reasoning
Chain-of-Thought (CoT) prompting (Wei et al. 2023) requires no retraining: a worked example or a request to “think step by step” elicits intermediate text. Each new token can condition on those written steps, but the model can also continue a mistaken argument. Giving PaLM 540B eight CoT examples raised GSM8K accuracy from 18% to 57% in the reported experiment, entirely through a changed prompt.
Reasoning need not be linear. Tree-of-Thought extends the chain-of-thought concept by allowing exploration of multiple reasoning paths simultaneously. Instead of following one chain, this method lets the model branch into multiple reasoning paths, score partial solutions, and expand on the ones that look promising. It can backtrack when a path fails. In the “Game of 24” puzzle, GPT-4 combined with CoT reasoning solved only 4% of the puzzles, but replacing it with ToT raised the success rate to 74% (Yao et al. 2023). The comparison spends additional inference compute, whose value depends on the evaluator and search budget.
Chain-of-Action (CoA) (Zhenyu Pan et al. 2025) interleaves reasoning and external tool use. This approach decomposes a complex query into a reasoning chain interleaved with tool calls such as web search, database lookup, or image retrieval that are executed on the fly and fed into the next thought. Tool observations can provide new evidence, but the source can be stale, irrelevant, or wrong. For Alder, fetching P2 and checking the order date supplies information that a longer unsupported explanation cannot. CoA outperformed leading CoT and RAG baselines by approximately 6% on the authors’ multimodal QA benchmarks, particularly on compositional questions that need both retrieval and reasoning. The broader tool loop is developed in Chapter 25.
Reflexion
Reflexion (Shinn et al. 2023) introduces a self-improvement mechanism that operates through iterative feedback. After each attempt, the model writes a short reflection on what went wrong or could be improved. That remark is stored in memory and included in the next prompt, giving the model a chance to revise its approach on the next try. When feedback identifies a real error and the revision uses it, the model gains short-term memory and avoids repeating past mistakes. This is application-managed context, not a weight update. The study reports a GPT-4 agent raising HumanEval success from 80% to 91%. Without useful feedback, critique can replace a correct answer with a wrong one; check revisions against evidence or an independent verifier.
Test-time Compute
Training a reasoning policy changes weights; giving that policy more time, samples, or tool calls at test time changes the computation spent on an answer. Search can backtrack over stored candidates and revise an answer, even though each autoregressive continuation appends tokens sequentially.
Extended thinking budgets illustrate this distinction. Figure 24.4 records a historical experiment with Claude 3.7 Sonnet on AIME 2024, varying the available thinking budget.
Figure 24.4 shows accuracy increasing logarithmically with actual thinking-token usage in the reported experiment. The model often stopped before its maximum budget; this alone does not establish that it knew when its reasoning was sufficient. Source: Anthropic’s Visible Extended Thinking.
Other ways to spend test-time compute include parallel candidate generation and dynamic tool selection. In s1, budget forcing extends a trace when the model would otherwise stop (Muennighoff et al. 2025). Compare these interventions under the same token, tool-call, and latency limits: additional computation can improve a result, repeat the same error, or introduce a new one.
What a Reasoning Trace Shows
One form of reasoning combines independent facts. In a study of Claude, Anthropic traced a response to “What is the capital of the state where Dallas is located?” through features associated with “Dallas is in Texas” and “the capital of Texas is Austin.” This is evidence about a studied computation, rather than a claim that every answer follows such a path.
This multi-step reasoning process can be visualized as follows:
Intervening on the intermediate representation provides a stronger check than reading a fluent explanation. Replacing the Texas-related feature with a California-related feature shifted the answer toward Sacramento. Such interventions support a causal role for that representation in the studied example.
A written chain of thought need not faithfully describe the computation that produced the answer. Anthropic’s examples include both a trace consistent with identified intermediate calculations and an incorrect hinted answer followed by a plausible justification.
For the square root of 0.64, the studied features included intermediate quantities such as the square root of 64. With a difficult problem and a false hint, the model could instead generate steps supporting the hinted conclusion. The trace alone does not distinguish these cases.
Next-token training does not install a conventional calculator, but networks can learn arithmetic computations. For addition such as 36+59, the study identified interacting approximate-magnitude and precise last-digit computations.
This is neither evidence for a memorized table of every sum nor proof that the model uses schoolbook addition. Interpretability examines particular learned mechanisms; an external calculator supplies a separate way to check the result.
Figure 24.8 contrasts the two cases: a faithful trace for the square root of 0.64, and a fabricated one for the cosine of a large number. For high-stakes uses, the reasoning trace has to be verified, not taken at face value.
24.5 Dealing with Context Window Limitations: Context Engineering
A longer context window allows more evidence, but does not establish that the model will use it correctly. For Alder, both P1 and P2 may fit in the prompt; deciding which governs the order remains necessary.
A conventional fixed-weight model does not automatically retain a previous request. A context window limits the tokens available to a call, subject to the model’s input and output limits. Applications create continuity by storing and resupplying history or retrieving records.
Context management balances cost, latency, and accuracy. More evidence can help, but irrelevant passages consume budget and create opportunities for confusion.
A useful mental model treats the context window as a budget:
\[ S + T_{\mathrm{tools}} + H + R + U + O \leq B \]
where \(B\) is the model-specific usable budget, \(S\) is system instructions, \(T_{\mathrm{tools}}\) is tool schemas, \(H\) is conversation history, \(R\) is retrieved evidence, \(U\) is current input, and \(O\) reserves generated output, including reasoning tokens where these count toward the limit. Separate API input and output limits must also be respected. Context engineering decides what to keep, compress, retrieve, and leave available for the answer.
Retrieval-Augmented Generation
Retrieval-Augmented Generation (RAG) retrieves relevant documents and inserts them into the prompt. It can supply facts unavailable or outdated in the weights, without changing those weights. For Alder, retrieve the policy applicable to the order date, then ask the assistant to answer with the policy identifier and supporting passage.
For dense retrieval, an embedding model converts the query and document chunks into vectors; a search system compares them and selects candidates. The retrieved passages accompany the original request in the generation prompt. Retrieval selects evidence; the generator still has to interpret it.
On an enterprise workflow-generation task, RAG reduced hallucinated steps and tables from 21% to 7.5% (Ayala and Bechard 2024). This is a task-specific result. Retrieved multilingual documents can also supply evidence for answers across languages, provided the relevant text is retrieved and used correctly.
Suppose P2 is retrieved correctly, yet the assistant answers, “Returns after 14 days are allowed with a receipt.” P2 contains no such exception. A citation to P2 does not support the added claim. If P1 was retrieved instead, the failure is evidence selection; if P2 was supplied and misused, the failure is generation. The two require different repairs.
A basic RAG pipeline:
- Index: Split documents into chunks, embed into vectors, store in vector database
- Retrieve: Given a query, find similar chunks via embedding similarity
- Generate: Concatenate retrieved chunks with the query and prompt the LLM
The index can cover a corpus larger than the context window, while only selected evidence enters each call.
RAG versus Fine-tuning
RAG and fine-tuning (discussed in Section 24.4.1) represent two distinct strategies for adapting LLMs to specific applications. While fine-tuning modifies the model’s internal parameters based on domain-specific data, RAG retrieves relevant information from external knowledge bases and incorporates it into the prompt at inference time. In RAG, the model’s parameters remain unchanged during the retrieval process.
RAG is useful when information changes frequently, labeled data are limited, or the answer must cite external sources.
Fine-tuning is useful when labeled examples define recurring behavior, format, or domain conventions. Required compliance still needs evaluation and external enforcement where appropriate.
Retrieval-Augmented Fine-Tuning (RAFT) specifically trains on tasks that require using retrieved documents, including relevant evidence and distractors. Merely using RAG alongside an independently fine-tuned model does not establish that it was trained this way.
When to Use Long Context vs. RAG vs. Summarization
The task suggests a starting comparison:
| Scenario | Strategy | Rationale |
|---|---|---|
| Document fits in window; need holistic reasoning | Supply the full document | Cross-references and global structure preserved |
| Corpus larger than window; factual lookup | RAG with small chunks | Precision matters more than narrative |
| Corpus larger than window; synthesis task | RAG with hierarchical retrieval | Need both global context and specific evidence |
| Long conversation history | Tiered memory + summarization | Keep recent turns verbatim; compress older context |
| Repetitive prompt structure | Caching + compression | Avoid paying to reprocess static content |
The decision often comes down to whether the task requires holistic reasoning (favor long context) or precise evidence retrieval (favor RAG). Many production systems combine both: retrieve evidence via RAG, then reason over that evidence within the remaining budget.
A Policy-corpus Ablation
The following is a proposed experiment, not a report of measured gains. Extend Alder’s corpus with the same policy documents for other regions and unrelated product rules. Reserve an illustrative 4,096-token evidence allowance within the input budget and 1,024 tokens for output. Use one checkpoint, one answer prompt, greedy decoding, and no retries for every condition; record both time to first token and total response time.
| Condition | Evidence supplied | Failure to inspect |
|---|---|---|
| Full context | All documents, if they fit the allowance | Conflicting versions or buried evidence |
| Basic retrieval | Highest-scoring chunks within the allowance | Similar but obsolete P1 outranks P2 |
| Metadata filtering | Filter by region and applicability date before retrieval | Incorrect or missing document metadata |
| Filtering plus reranking | Score query-passage pairs jointly after retrieval | A relevant passage is discarded or an unsupported exception is generated |
Freeze questions and expected answers before tuning retrieval. Compare correctness and unsupported-answer rates on the same questions, including return deadlines, policy transitions, and the unanswerable opened-goods case. Inspect retrieved passages as well as final answers. A reranker cannot recover P2 if the initial candidate set omitted it. The evaluation protocol in Section 24.6 separates retrieval failures from failures to use sufficient evidence.
The Lost-in-the-Middle Problem
In the multi-document experiments of N. F. Liu et al. (2024), accuracy often followed a U-shaped curve: evidence at the start or end was easier to use than evidence in the middle. GPT-3.5-Turbo’s accuracy could drop by more than 20% as the gold document moved toward the middle, at times below its 56.1% closed-book accuracy. This is evidence from specified models and tasks.
For a policy corpus, test whether the answer changes when P2 moves within an otherwise fixed prompt. Positioning is one possible mitigation, alongside removing distractors and reducing the retrieved set. Its benefit must be measured for the chosen model.
Improving Evidence Selection
Chunking defines the units retrieval can find. Fixed-size chunks may sever a rule from its exception; semantic chunking groups sentences using similarity. A greedy threshold method adds a sentence while it remains similar to the current chunk. Chroma’s ClusterSemanticChunker instead uses dynamic programming and reported better token-level precision and intersection over union than the alternatives in its evaluation (Smith and Troynikov 2024). For Alder, keep the return condition, effective date, and policy identifier together. Chunk size is a parameter to test, not a universal lookup or narrative default.
Pure vector search can miss exact tokens, identifiers, or acronyms. A user searching for “RFC 2616” needs lexical matching, not semantic similarity to “HTTP specification.” Hybrid retrieval combines dense embeddings with keyword search (BM25). Results merge via a weighted combination of their normalized scores:
\[ H = (1 - \alpha) \cdot \text{BM25 Score} + \alpha \cdot \text{Semantic Score} \]
A common starting point is equal weighting, \(\alpha\approx0.5\), as in the retrieval-metrics computing exercise of this chapter; tune it after normalizing score scales. A larger weight on dense retrieval is not automatically better. See Weaviate’s discussion.
Many retrieval failures concern scope. For “What is the refund policy for EU customers?”, use region and effective-date metadata to exclude inapplicable policies before similarity scoring. An order under P1 may still need P1, so “use the newest document” is insufficient. See Haystack’s metadata filtering guide.
Vector similarity is a coarse filter. A cross-encoder reranker scores query-passage pairs jointly. Late-interaction models such as ColBERT (Khattab and Zaharia 2020; Santhanam et al. 2022) retain token-level representations for finer matching. Candidate count and the final evidence count trade recall, precision, cost, and latency; compare them within the ablation budget.
For large interconnected corpora, hierarchical retrieval can retrieve summaries and their source passages. RAPTOR builds a tree of summaries (Sarthi et al. 2024); GraphRAG uses extracted entities, relationships, and community summaries for corpus-level questions (Larson and Truitt 2024). They add indexing and summarization costs that the small Alder corpus may not justify.
| Variant | Mechanism | Intended use |
|---|---|---|
| Self-RAG | Learned retrieval and self-critique decisions | Adaptive evidence use |
| CRAG | Adaptive retrieval with confidence evaluation and web fallback | Tasks requiring current information |
| Graph RAG | Knowledge graph extraction; uses graph-based summaries | Multi-entity reasoning |
| HyDE | Generates hypothetical answer, retrieves based on that | Vague or ambiguous queries |
HyDE generates a hypothetical answer and retrieves using its embedding (Gao et al. 2022). This can bridge a vocabulary mismatch, but the hypothetical answer is a search aid, not evidence. CRAG adds retrieval evaluation and corrective retrieval; Self-RAG trains the model to decide when to retrieve and critique its outputs. These address different failure mechanisms.
Mitigating Lost-in-the-Middle
Mitigation can change prompts and retrieval (black-box methods) or model architecture and attention (white-box methods). The GM-Extract study (Gupte et al. 2025) separates document retrieval from variable extraction and finds context-dependent benefits. Preserve that distinction when evaluating a method.
One positioning heuristic puts the most relevant passage first, another highly relevant passage last, and weaker candidates between them. It changes placement, not relevance scores, and should be compared with the same passages in their original order.
OpenAI’s December 2025 GPT-5.2 report uses MRCRv2 to test long-context retrieval: repeated identical requests are inserted among similar exchanges, and the model must reproduce the response to a specified occurrence. The metric is mean string-match ratio, reported by input-length bin at maximum reasoning effort. In the eight-needle task at 128k to 256k input tokens, OpenAI reports 77.0% for GPT-5.2 against 29.6% for GPT-5.1; MRCRv2 also corrects about 5% of tasks with incorrect answers in the first version. This tests a narrower behavior than general conversational coherence (OpenAI’s GPT-5.2 report).
Prompting the model to cite each relevant policy clause can make omissions easier to inspect, but does not force it to attend correctly to every chunk. Check the cited passage and test whether moving it changes the answer.
Joint long-context reranking would score related passages together. The configuration below instead expands retrieved sentences to their local windows and reorders results; it contains no learned joint reranker and demonstrates no accuracy improvement. It is an optional library illustration, assuming documents and the model configuration have already been supplied. The window size and retrieval count are illustrative.
Caching and Compression
Prompt compression reduces input tokens. LLMLingua-2 uses token classification and reports 2–5× compression with limited performance loss in its evaluated settings (Zhuoshi Pan et al. 2024). Check downstream accuracy after compression: dropping a negation from a return rule or changing user_id to userid can invalidate the result. Verbose prose, repeated retrieval passages, and repeated reasoning context are possible candidates, while schemas and code demand exact preservation.
Context caching reuses computation for unchanged prompt prefixes, such as system instructions or tool schemas. Anthropic’s prompt caching is one implementation. It can reduce repeated prefill work and charges under the provider’s rules, but does not enlarge the usable evidence window or make stale instructions correct.
Even with enough tokens, context can fail. Instruction conflicts create incompatible directions; retrieval drift supplies related but unhelpful passages; duplication wastes budget; stale summaries lose earlier constraints; prompt injection inserts malicious instructions into retrieved material. Conflict rules, deduplication, source checks, and constrained tool access address different parts of these failures.
Tiered application memory keeps recent turns verbatim, compresses older history into session summaries, and retrieves persistent records on demand. Alder might retain the order date and policy identifier across turns. The summary must not silently replace that date or retain P1 after evidence establishes that P2 applies.
The MemGPT pattern virtualizes LLM context, treating it like an operating system’s page cache (Packer et al. 2023). At 70% of capacity, reasoning pauses for memory-pressure handling: the system identifies the least critical content, compresses it to an external tier, and retrieves relevant context back on demand. Conversation can continue beyond one window, but this is not lossless retention: omissions, retrieval misses, and stale summaries remain possible.
What “Memory” Means
| State | Location and update | Lifetime | How it is accessed | Failure mode |
|---|---|---|---|---|
| Conversation history | Application stores turns | Configured session | Resupplied in the prompt | Truncation or excess tokens |
| Session summary | Application writes a compressed record | Until replaced or deleted | Inserted into later prompts | Lost constraints or stale facts |
| Persistent records | External database is written or revised | Across sessions if retained | Retrieval | Wrong record or obsolete policy |
| KV cache | Server stores attention keys and values | While the prefix is retained | Direct reuse of computation | Growing memory use or cache invalidation |
| Neural memory | A specified architecture updates internal state | Defined by its retention and reset rules | Learned memory access | Interference or forgetting |
These states have different lifetimes and update rules. A KV cache accelerates attention to tokens already present; an external record can outlive a request; neither automatically updates the base model’s learned weights.
Evaluation and Deployment
Retrieval quality separates evidence selection from answer generation. Precision@k is the fraction of the top-\(k\) retrieved chunks that are relevant; Recall@k is the fraction of all relevant chunks retrieved among those \(k\); Mean Reciprocal Rank averages \(1/\text{rank}\) for the first relevant result (zero if none is found). Compression ratio is original tokens divided by compressed tokens, but task performance after compression matters more than the ratio alone. Generation can be evaluated with exact match, token-level F1, and faithfulness to supplied evidence. RAGAS supplies automated metrics, whose judgments also need validation.
Enterprise assistants can combine hybrid retrieval and metadata filtering; research systems may need hierarchical or graph-based summaries; code assistants can combine symbol and filename lookup with dense search. Treat retrieval counts, compression ratios, and latency targets as application settings to measure on a stated corpus, model, and hardware, rather than transferable defaults.
In the Alder ablation, report whether P2 was retrieved before scoring the final answer. That error slice distinguishes a missing-evidence problem from ignoring sufficient evidence. Monitor retrieval precision, token use, and answer quality together as policies change.
24.6 Evaluating Model Performance
Use the held-out Alder questions to choose between systems, rather than using a leaderboard rank as the decision. Freeze the questions, expected policy version, allowed evidence, prompt, decoding settings, retry policy, and compute budget. For a serving comparison, change the checkpoint while holding this protocol fixed; for the context ablation, change only evidence selection.
Correctness is the fraction of questions answered according to the applicable policy, counting abstention as correct on cases designated unanswerable. Report unsupported-answer rate as the fraction of all questions receiving an answer containing at least one unsupported factual claim. Report abstention separately on answerable and unanswerable cases so that refusing everything cannot look successful. Record time to first token, total response latency, and total expenditure divided by successful tasks, including expenditure on failed attempts. If there are no successes, cost per success is undefined.
Compare paired outcomes question by question. For small score differences, report uncertainty, for example a paired bootstrap interval resampling questions, and inspect error slices such as policy transitions or missing evidence. Repeat stochastic runs when sampling is part of the system. Preference measures what raters favor; benchmark accuracy measures a specified target; faithfulness measures support from supplied evidence. A faithful answer based on an obsolete document can still be factually wrong for the order.
Public benchmarks supply additional task families. GPQA Diamond uses graduate-level science questions; ARC-AGI uses abstract pattern tasks; AIME uses competition mathematics; MMMU-Pro uses questions requiring joint reasoning over text and images. Their scores measure performance under particular protocols, not general intelligence or reliability on Alder’s policies.
SWE-bench evaluates resolving repository issues, reporting the percentage of instances resolved.
Benchmarks earn their place because they are shared, fixed and cheap to rerun: the same questions scored the same way let anyone reproduce a claim, compare systems built by different teams, and track progress over years. Their weaknesses follow from the same properties. A fixed test set can be optimized directly, so a rising score may reflect tuning to the test rather than a more capable model, which is Goodhart’s law again. A single number hides the protocol behind it (benchmark variant, agent harness, tools, prompts, retries, inference budget), and two reported scores are comparable only when that protocol matches. Many benchmarks saturate within a few years of release, after which they no longer separate the strongest systems, and their tasks rarely match the task a deployment cares about. Leaderboard snapshots, such as the two below, therefore date quickly: read them as a record of one moment and one measurement protocol, not as a ranking that transfers to your application.
Benchmark contamination can inflate scores if training includes test material. LiveBench refreshes questions to reduce this risk. Humanity’s Last Exam collects difficult expert questions; Figure 24.9 is a January 2026 snapshot, not a current ranking. Its exact run configurations and uncertainty are not documented here.
LMArena collects blind pairwise preferences between responses to the same prompt and fits comparative ratings. The method is the Elo system used to rank chess players: a win raises a model’s score and a loss lowers it; beating a higher-rated opponent raises it more than beating a lower-rated one. Like chess ratings, they summarize relative success in comparisons, but user preference is not factual verification. Figure 24.10 records the WebDev category in January 2026; voting populations, uncertainty, and configurations matter when interpreting small rank differences.
Use these resources to identify candidates, then compare them on the held-out application task. Archive the checkpoint version, harness, prompts, evidence, raw outcomes, token counts, price date, and latency definition. Without those records, a screenshot illustrates a historical comparison but cannot establish a deployable winner.
24.7 Choosing the Right Model for Your Application
Bigger models are not always the better choice. Four factors drive model selection: data privacy, cost, task-specific performance, and latency.
Data privacy. Data sensitivity, regulations (HIPAA, GDPR), contractual requirements, and security policy constrain deployment. Hosting an open-weight model on controlled hardware gives control over data handling but incurs hardware and maintenance costs. An isolated cloud service is another option, with network isolation and retention determined by the actual service configuration and contract.
Cost. Compare total serving expense, including hardware utilization, operations, retrieval, and retries. A smaller hosted model may be economical at sufficient utilization; a low-volume service may favor an API. Measure cost per successful task under the same protocol.
Task-specific performance. Very small models (around 3B parameters or fewer) run on consumer hardware for tasks like text classification or domain-specific chatbots; medium models (7 to 30B) are often the sweet spot; large models (30B+) give the best general performance but require specialized hardware. Code, multilingual, and domain variants offer different capabilities. Parameter count alone does not rank them. Distillation, fine-tuning, and quantization can help, but each adapted system needs evaluation.
Latency. Time to First Token (TTFT) measures the delay before output begins; total response time includes generation. Speech bots (Alexa, Google Home) and real-time applications such as trading may prioritize TTFT, while a tool workflow may wait for a complete structured answer. Record both under representative load.
Scale and Its Limits
For decades, high-quality data analysis required parsimonious models, as small as possible while still capturing essential patterns. Deep learning broke this trend: models work very well when the number of parameters is large, often exceeding the number of data points. We discussed this phenomenon in Section 19.8.
Published model sizes grew rapidly: GPT-1 (2018) had 117 million parameters. GPT-2, with 1.5 billion parameters, was initially released in stages over several months because of misuse concerns, a caution OpenAI later relaxed after independent analysis found the risks manageable. GPT-3 had 175 billion. The selected models in Figure 24.11 give a historical view of this growth.
Figure 24.11 shows published sizes of selected models (Radford et al. 2018, 2019; Devlin et al. 2019; Raffel et al. 2020; Brown et al. 2020; Rae et al. 2021; Hoffmann et al. 2022; Chowdhery et al. 2022; Touvron et al. 2023; Grattafiori et al. 2024; DeepSeek-AI et al. 2024). Sizes grew by nearly four orders of magnitude from 2018 to 2022; the plotted later models lie within about one order of magnitude. This selection does not measure where subsequent compute was spent. Chinchilla separately studies the trade-off between parameter count and training tokens at fixed compute (Hoffmann et al. 2022).
Mixture of Experts (MoE) models activate a subset of their parameters for each token. Total parameters measure stored capacity; active parameters describe the selected computation more closely. Neither alone determines total serving cost.
Multimodal Transformers already process text, images, audio, and other modalities, though architectures and supported inputs and outputs differ. They extend the same model-selection question: which system meets the task’s quality and resource constraints?
| Quantity | What it measures | What it leaves unspecified |
|---|---|---|
| Total parameters | Stored model capacity | Parameters active for one token |
| Active parameters | Weights used for a token | Sequence length and hardware efficiency |
| Training tokens | Exposure during learning | Data quality and optimization choices |
| Inference compute | Work spent answering a request | Correctness of the resulting answer |
Quantization and Serving Efficiency
Distillation replaces a model through teacher-student training; fine-tuning adapts its learned behavior; quantization changes numerical representation. These interventions can be combined, but quantization is not itself instruction tuning.
Quantization reduces the precision of weights and sometimes activations, for example from 16-bit to 4-bit representations. It reduces weight-storage requirements and memory traffic, while arithmetic speed depends on supported kernels and hardware. GPTQ and AWQ choose quantization schemes to limit task degradation; evaluate the quantized model rather than assuming unchanged accuracy.
The mathematics of quantization involves mapping continuous values to a discrete set of levels. For a weight \(w\) with range \([w_{min}, w_{max}]\), 8-bit quantization maps it to one of 256 integer values: \[ w_q = \text{round}\left(\frac{w - w_{min}}{w_{max} - w_{min}} \times 255\right) \]
Post-training quantization uses calibration data to select scales or corrections for already trained weights. Quantization-aware training models quantization effects during optimization. Both require measuring the resulting quality and serving trade-off.
Speculative decoding (Leviathan, Kalman, and Matias 2023; Chen et al. 2023) uses a draft model to propose several tokens, which the target model scores in parallel. An exact rejection-sampling rule accepts a prefix; at the first rejection it samples a replacement from a corrected distribution and discards later draft tokens (each was conditioned on the rejected token, so none can be salvaged). If all proposals are accepted, the target supplies an additional token. Under this rule, the output distribution matches that of target-only sampling, so the reported 2 to 3x speedups cost nothing in output quality. Speed depends on draft cost, acceptance rate, and hardware; distribution preservation is not a factual-accuracy guarantee.
KV caching stores attention keys and values for preceding tokens. For dense attention with fixed layer width, decoding a new token against \(n\) cached tokens costs \(O(n)\) attention work, with \(O(n)\) growing cache memory. Prefilling an \(n\)-token prompt still requires \(O(n^2)\) attention work, and generating a sequence of growing length still has quadratic total attention cost. Caching avoids recomputing all prefix representations at every decoding step; it does not make total generation linear.
Streaming prefill (or incremental prefill) starts processing input before the entire query is available. The model begins the “prefill” phase (processing input and building the KV cache) on chunks as they arrive, useful for voice-activated systems where a command can be processed while the user is still speaking. Reuse requires an unchanged prefix; transcript revisions can invalidate cached work.
Continuous batching (Yu et al. 2022) lets the server start processing new requests immediately, even if other requests in the same batch are mid-generation. Unlike static batching, which waits for all sequences to finish, continuous batching dynamically inserts and removes requests, increasing throughput and reducing wait times.
Attention Efficiency: FlashAttention and Ring Attention
Transformers rely on attention, and naive attention scales quadratically with sequence length (\(O(n^2)\)). Even with optimizations, long prompts increase memory traffic and reduce throughput.
FlashAttention tiles exact attention to reduce transfers between GPU memory and on-chip storage, avoiding materialization of the full attention matrix (Dao et al. 2022). Its attention-memory use scales linearly with sequence length for fixed width, while arithmetic work remains quadratic. FlashAttention-3 further optimizes these kernels for H100 GPUs, reaching up to 740 TFLOPs/s (Shah et al. 2024); kernel throughput is not the memory footprint or latency of an entire LLM.
Ring Attention distributes sequence blocks across devices, exchanging key and value blocks in a ring while computing attention (H. Liu, Zaharia, and Abbeel 2023). Under its sharding assumptions, available context can grow with device count, with communication and total compute costs still to account for.
These optimizations change execution efficiency, not the evidence or learning objective.
The Rise of Small Language Models
Distillation and efficient fine-tuning have contributed to the rise of Small Language Models (SLMs). These models, typically ranging from a few hundred million to several billion parameters, challenge the assumption that bigger is always better.
Possible benefits include local deployment, reduced energy use, and shorter latency, subject to hardware and workload. Privacy also depends on the surrounding application’s data handling.
A possible workflow distills teacher behavior, adapts the student with LoRA, then quantizes it. Test each step for gains and regressions. As a historical product comparison, Google reported Gemini 3 Flash at 78% on SWE-bench Verified against Pro’s 76.2% in late 2025 (Google 2025). The reports do not establish a teacher-student lineage or relative parameter counts; they illustrate that a faster product can be competitive on a particular task.
24.8 Research Outlook: Neural Memory Systems
Neural memory research changes the architecture itself. Titans and MIRAS (Behrouz, Zhong, and Mirrokni 2025; Behrouz et al. 2025) study memory modules updated during sequence processing, rather than only external records retrieved into a conventional prompt. Like recurrent models, they compress earlier inputs into updated internal state, connecting recurrent memory with attention.
Titans uses a learned MLP memory module updated at test time through a gradient-based surprise signal, combined with attention. Linear RNNs such as Mamba store memory in low-rank structures; Titans reports that a deep MLP memory captures structure that shallow compression cannot. It reports experiments on language modeling, reasoning, and other sequence tasks, including long-context retrieval. Its memory-depth experiments include 760M-parameter models evaluated on sequence perplexity, not durable personal memory across sessions.
MIRAS organizes associative-memory designs by architecture, learning objective, retention rule, and optimizer. It interprets forgetting through retention regularization and studies alternatives to squared-error memory objectives, including L1 and Huber losses, which are less sensitive to large residuals: L2 overreacts to extreme tokens, while L1 or Huber gives more stable behavior. The equivalence between RNN forget gates and weight decay reframes forgetting as a controllable design parameter. The evidence concerns experimental sequence models.
Larimar (Das et al. 2024) adds episodic memory to a language model for rapid knowledge updates. Its reported 8–10× speedups concern knowledge-editing comparisons.
Persistence depends on which state is saved, how it is retrieved, and when it is reset. For Alder, an external policy record, a session summary, and an experimental neural memory have different failure modes. Each must still supply the applicable policy and preserve its conditions.
24.9 Combining Techniques
In practice, interventions stack. Most large LLMs, including GPT-4, are first trained with supervised fine-tuning and then polished with RLHF. Alder’s assistant can learn a response format through SFT, retrieve P2 at inference, check the order date with a tool, and serve the resulting model with quantized weights and cached prefixes. The reason for each change should be a measured failure or resource constraint.
| Ways to improve an LLM system | Analogy | Weight change? | Signal or mechanism | Policy-assistant use |
|---|---|---|---|---|
| Instruction or domain SFT | Flashcards, medical school specialization | Yes, weights or adapters | Demonstrated responses | Format, vocabulary, and abstention examples |
| Distillation | A student learning from a teacher | Yes, student weights | Teacher outputs | A less expensive task model |
| RLHF or DPO | Teacher feedback | Yes | Ranked responses | Preferred behavior, subject to factual checks |
| Verifier-based RL | Checking against an answer key | Yes | Checkable outcomes | Train on tasks with a defensible verifier |
| CoT or Tree-of-Thought | Showing your work, exploring a decision tree | No | Intermediate text or search | More computation with a fixed policy |
| Reflexion | Learning from mistakes | No | Stored critique and feedback | Revise after a diagnosed failure |
| RAG | An open-book exam | No | Retrieved passages | Supply the applicable policy |
| Chain-of-Action or tool calls | Using tools while thinking | No | External observations | Fetch evidence or check dates |
| Quantization and caching | Rounding numbers to save space | Stored values or execution, not learned behavior | Lower precision or reused computation | Reduce serving resources |
Summary
Next-token likelihood explains what the model learns; decoding explains how it generates an answer. Supervised, preference, and verifier-based training use different signals to change behavior. Retrieval and tools can add evidence without changing weights, while serving optimizations change resource use.
Return to the incorrect refund answer. If the assistant retrieved P1 for a P2 order, repair evidence selection. If it invented an exception despite receiving P2, test evidence use and answer checks. If the policy says nothing about opened goods, abstention is appropriate. Compare each repair on held-out questions under the same budget, including its unsupported answers, latency, and cost.
Exercises
Pen-and-Pencil
Exercise 24.1 (Autoregressive Generation) An LLM outputs logits for the next token: \(z = (z_1, z_2, z_3, z_4, z_5) = (3.2, 2.5, 2.0, 1.5, 0.5)\) corresponding to tokens (mat, floor, chair, table, dog).
- Compute the softmax probabilities at temperature \(T=1\).
- Compute the softmax probabilities at temperature \(T=0.5\) and \(T=2.0\). How does the distribution change?
- Using greedy decoding, which token is selected? Does the choice depend on temperature?
- Define nucleus (top-\(p\)) sampling. For \(p=0.8\) at \(T=1\), which tokens are in the nucleus?
- Compute the entropy \(H = -\sum_i p_i \log p_i\) at each temperature. What is the relationship between temperature and entropy?
Exercise 24.2 (Autoregressive Probability) An LLM generates text by factoring the joint probability using the chain rule: \[ p(x_1, x_2, \ldots, x_n) = \prod_{t=1}^{n} p(x_t \mid x_1, \ldots, x_{t-1}) \]
- Show that this factorization holds for any joint distribution (no independence assumptions needed).
- A model assigns the following conditional probabilities to generate “the cat sat”: \(p(\text{the}) = 0.15\), \(p(\text{cat} \mid \text{the}) = 0.08\), \(p(\text{sat} \mid \text{the cat}) = 0.12\). What is the joint probability of this sequence?
- Define perplexity as \(\text{PPL} = \exp\left(-\frac{1}{n}\sum_{t=1}^n \log p(x_t \mid x_{<t})\right)\). Compute the perplexity of the sequence in (b).
- A competing model assigns probabilities 0.12, 0.10, 0.15 to the same three tokens. Which model assigns higher probability to this sequence? Which has lower perplexity?
Exercise 24.3 (LoRA: Low-Rank Adaptation) In LoRA, a pretrained weight matrix \(W \in \mathbb{R}^{d \times k}\) is updated as \(W' = W + BA\) where \(B \in \mathbb{R}^{d \times r}\) and \(A \in \mathbb{R}^{r \times k}\) with rank \(r \ll \min(d,k)\).
- For a transformer layer with \(d = k = 4096\), compute the number of trainable parameters for LoRA ranks \(r \in \{1, 4, 8, 16, 64\}\). Express each as a percentage of the full \(d \times k\) parameters.
- If LoRA is applied to the query and value projection matrices in each of 32 transformer layers, how many total trainable parameters for \(r = 8\)?
- Why is \(A\) typically initialized with a random Gaussian and \(B\) initialized to zero? What would happen if both were random?
- Suppose a full fine-tuning achieves test accuracy 94.2% and LoRA with \(r=8\) achieves 93.8%. Is the 0.4% gap worth the parameter savings? Argue both sides.
Exercise 24.4 (Knowledge Distillation) A teacher model produces soft targets \(q_i = \exp(z_i/T) / \sum_j \exp(z_j/T)\) for a 4-class classification with logits \(z = (5.0, 2.0, 1.0, 0.5)\).
- Compute the soft targets at \(T=1\) and \(T=4\). Why are higher-temperature targets more informative for the student?
- The distillation loss combines soft and hard targets: \(L = \alpha T^2 \cdot \text{KL}(q^T \| p^T) + (1 - \alpha) \cdot \text{CE}(y, p^1)\), where \(q^T\) and \(p^T\) are teacher and student distributions at temperature \(T\), and \(y\) is the true label. Why is the \(T^2\) factor needed?
- If the true label is class 1 (the correct answer), explain what information the soft targets provide beyond the hard label.
Exercise 24.5 (Quantization) A neural network layer has weights \[ w = (0.73, -1.24, 0.05, 2.31, -0.89). \]
- Apply 8-bit linear quantization: \(w_q = \text{round}\left(\frac{w - w_{\min}}{w_{\max} - w_{\min}} \times 255\right)\). What are the quantized integer values?
- Dequantize back to floating point using \(\hat{w} = w_q \times \frac{w_{\max} - w_{\min}}{255} + w_{\min}\). What is the maximum absolute quantization error?
- A 7B parameter model uses 16-bit floats. How much memory does it require? How much with 8-bit quantization? With 4-bit?
- Why might quantization degrade performance more on mathematical reasoning tasks than on text summarization?
Exercise 24.6 (Instruction Fine-Tuning: Loss Masking and Domain Adaptation) Instruction fine-tuning (IFT) runs a second pass of supervised learning on (instruction, input, answer) triples such as the “solve this math word problem step by step” example in Section 24.4. Under teacher forcing a causal language model computes a next-token cross-entropy loss \(\ell_t\) at every position of the tokenized triple; IFT differs from ordinary continued pretraining only in which positions contribute to the loss.
A tokenized example has 3 instruction tokens (per-token losses \(0.2, 0.2, 0.2\)), 3 input tokens (losses \(0.4, 0.4, 0.4\)), and 2 answer tokens (losses \(3.0, 1.0\)). Compute (i) the full-sequence loss \(L_{\text{full}} = \frac{1}{n}\sum_{t=1}^{n} \ell_t\) used by ordinary continued pretraining, and (ii) the answer-only masked loss \(L_{\text{IFT}} = \frac{1}{|A|}\sum_{t \in A} \ell_t\) used by instruction tuning, where \(A\) is the set of answer-token positions. By what factor do they differ?
Explain why IFT masks the loss on the instruction and input tokens instead of including them. What would the model be optimized to do if they were included, and why is that undesirable given that at inference time the instruction and input are supplied by the user rather than generated by the model?
In ICD-10 medical coding, domain supervised fine-tuning raised exact-code accuracy from under 1% at the pretrained baseline (take 0.5% as a representative value) to about 97% on standard coding tasks, but only 69% on real clinical notes (Hou et al. 2025). Compute the multiplicative improvement factor \(97/0.5\) and the retention ratio \(69/97\). What does the gap between these two numbers reveal about what IFT does and does not fix?
Section 24.4.3 describes RLHF’s first stage as supervised fine-tuning “on high-quality demonstrations of desired behavior.” Is this the same computation as the masked loss in part (a)? What differs, in purpose and in training data, between generic instruction tuning (Flan-style, a broad task mixture whose 540B model showed an average gain of 8.9 points across MMLU, BBH, TyDiQA, and MGSM, Chung et al. (2022)) and the supervised fine-tuning stage that precedes reward-model training in RLHF?
Exercise 24.7 (Context Window Budget) An LLM has a context window of 8,192 tokens. You are building a RAG-based customer support system with the following token requirements:
- System prompt: 500 tokens
- Tool descriptions (5 tools): 150 tokens each
- Conversation history: 3 prior turns averaging 200 tokens each
- Retrieved documents: up to \(k\) chunks of 300 tokens each
- User query: ~100 tokens
- Reserved for model response: 1,500 tokens
- Write the budget equation and solve for the maximum number of retrievable chunks \(k\).
- If you need \(k = 20\) chunks, which components would you reduce and by how much?
- The “lost-in-the-middle” phenomenon suggests that LLMs attend more to the beginning and end of the context. How should you arrange the retrieved chunks to mitigate this?
Exercise 24.8 (Direct Preference Optimization) The DPO loss for a pair of responses (preferred \(y_w\), dispreferred \(y_l\)) given prompt \(x\) is: \[ \mathcal{L}_{\text{DPO}} = -\log \sigma\left(\beta \left[\log \frac{\pi_\theta(y_w \mid x)}{\pi_{\text{ref}}(y_w \mid x)} - \log \frac{\pi_\theta(y_l \mid x)}{\pi_{\text{ref}}(y_l \mid x)}\right]\right) \] where \(\sigma\) is the sigmoid function, and \(\pi_{\text{ref}}\) is the frozen reference model.
- What happens to the loss when the model strongly prefers \(y_w\) over \(y_l\) (relative to the reference)? When it prefers \(y_l\)?
- What role does \(\beta\) play? What happens as \(\beta \to 0\) and \(\beta \to \infty\)?
- DPO eliminates the need for a separate reward model used in RLHF. Explain why, by showing that DPO implicitly defines a reward \(r(x,y) = \beta \log \frac{\pi_\theta(y \mid x)}{\pi_{\text{ref}}(y \mid x)}\).
- Why does DPO keep a reference model \(\pi_{\text{ref}}\)? What would go wrong without it?
Exercise 24.9 (True/False: LLMs)
GPT-style models predict the next token given all previous tokens (autoregressive generation).
Larger models always produce better outputs than smaller models on every task.
RLHF trains a reward model from human preference data, then optimizes the LLM to maximize the learned reward.
A model with perplexity 15 on a dataset assigns higher average probability to each token than a model with perplexity 25.
LoRA fine-tuning modifies every parameter in the pretrained model.
Quantizing a model from 16-bit to 4-bit reduces memory by 75% with no change in inference speed.
In RAG, the retriever and generator are trained jointly end-to-end.
Computing
Exercise 24.10 (Token-Level Text Generation) Build a simple character-level language model and use it to generate text.
- Load a text corpus (e.g., first 10,000 characters of a book). Build a bigram frequency table: for each character, count how often each next character follows it.
- Convert the frequency table to conditional probabilities. Generate 200 characters by sampling from the conditional distribution at each step.
- Extend to a trigram model (condition on the previous 2 characters). Generate 200 characters and compare the quality with the bigram model.
- Compute the perplexity of both models on a held-out portion of the text. Which model has lower perplexity?
Exercise 24.11 (Temperature and Sampling Strategies) Using a pretrained model’s output logits, explore how different sampling strategies affect text generation. Top-\(k\) sampling keeps only the \(k\) highest-probability tokens, renormalizes their probabilities to sum to one, and samples from that truncated distribution.
Take the logits of a vocabulary of ten tokens, token_1 to token_10, to be \[
4.5, 3.2, 2.8, 2.1, 1.5, 0.8, 0.3, -0.5, -1.2, -2.0,
\] stored in the vector logits.
- Implement softmax with temperature. Plot the probability distribution for \(T \in \{0.1, 0.5, 1.0, 2.0, 5.0\}\).
- Implement top-\(k\) sampling. For \(k=3\), what is the probability of each token being selected at \(T=1\)?
- Implement top-\(p\) (nucleus) sampling. For \(p=0.9\) at \(T=1\), which tokens are included?
- Sample 1000 tokens using each strategy: greedy, top-\(k\) (\(k=5\)), top-\(p\) (\(p=0.9\)), and pure sampling at \(T=1\). Plot histograms comparing the token frequency distributions.
Exercise 24.12 (Scaling Laws) Neural scaling laws predict that test loss decreases predictably as a power law with model size, dataset size, and compute, an empirical regularity documented across many orders of magnitude by Kaplan et al. (2020) and refined by Hoffmann et al. (2022), whose Chinchilla study showed that model size and training tokens should be scaled together for a fixed compute budget. A simplified form for model size is: \[ L(N) = L_\infty + \left(\frac{N_0}{N}\right)^\alpha \] where \(L_\infty\) is the irreducible loss, \(N\) is the number of parameters, and \(N_0, \alpha\) are constants.
Given the following data from training runs:
| Parameters (M) | Test Loss |
|---|---|
| 10 | 3.80 |
| 50 | 3.15 |
| 100 | 2.90 |
| 500 | 2.45 |
| 1000 | 2.28 |
| 5000 | 2.05 |
- Plot log(parameters) vs. test loss. Does the relationship look approximately linear on a log scale (after subtracting the irreducible loss)?
- Assuming \(L_\infty = 1.8\), fit the power law by linear regression on \(\log(L - L_\infty)\) vs. \(\log(N)\). Estimate \(\alpha\) and \(N_0\).
- Using your fit, predict the test loss for a 10B parameter model.
- What are the limitations of extrapolating scaling laws?
Exercise 24.13 (Retrieval Quality Metrics and Hybrid Search in RAG) Section 24.5 evaluates retrieval with Precision@k (fraction of the top-\(k\) retrieved chunks that are relevant), Recall@k (fraction of all relevant chunks that appear in the top \(k\)), and Mean Reciprocal Rank (MRR, here the reciprocal rank \(1/\text{rank}\) of the first relevant chunk). It also fuses dense and lexical search through the formula \(H = (1-\alpha)\cdot\text{BM25} + \alpha\cdot\text{Semantic}\), with a typical \(\alpha \approx 0.5\). This exercise works through these mechanics on a small worked example, distinct from the earlier Context Window Budget exercise, which covers token budgeting rather than retrieval quality.
A knowledge base has 6 chunks (A through F). For the query “What is the timeout setting for RFC 2616 keep-alive connections?” a retrieval system reports the scores below. Only chunks B and E actually answer the question: B states the number tersely using the exact RFC identifier and section number, while E restates the same fact in more natural, paraphrased language.
| Chunk | Semantic (cosine) | BM25 (lexical) | Relevant? |
|---|---|---|---|
| A | 0.55 | 1.0 | No |
| B | 0.42 | 9.5 | Yes |
| C | 0.20 | 0.5 | No |
| D | 0.62 | 1.5 | No |
| E | 0.81 | 3.0 | Yes |
| F | 0.15 | 2.0 | No |
Rank the chunks by semantic score alone. Compute Precision@3, Recall@3, and MRR.
Min-max normalize the semantic and BM25 columns separately to \([0,1]\), and explain why normalizing before fusing is necessary given the very different native scales of cosine similarity and BM25. Using the fusion formula with \(\alpha = 0.5\) on the normalized scores, rank the chunks again and recompute Precision@3, Recall@3, and MRR.
Compare (a) and (b). Which relevant chunk was rescued from outside the top 3, and why? Relate this to the chapter’s point that a user searching for “RFC 2616” needs lexical matching that pure semantic similarity to “HTTP specification” would miss.
Sweep \(\alpha \in \{0, 0.3, 0.5, 0.7, 1.0\}\) and report Precision@3 and Recall@3 at each value. Is the chapter’s “typical \(\alpha \approx 0.5\)” consistent with what you find? At what value of \(\alpha\) does retrieval quality degrade, and why, given chunk B’s scores?
Exercise 24.14 (Self-Consistency: Majority Voting over Reasoning Paths) Section 24.4 reports that giving PaLM 540B eight chain-of-thought (CoT) examples raised its GSM8K accuracy from 18% to 57% (Wei et al. 2023), a gain that came entirely from a better prompt. The chapter also stresses that reasoning need not be linear: Tree-of-Thought explores several branches instead of one, and Reflexion revises an answer after critiquing itself. Self-consistency is a third way to spend extra inference-time compute: draw several independent CoT samples for the same problem and return the most common (mode) final answer instead of trusting a single sample.
Suppose each sampled reasoning path independently reaches the correct answer with probability \(p\), and otherwise lands on one of \(m\) distinct wrong answers, chosen uniformly at random among the \(m\) options. Majority voting returns whichever of the \(m+1\) possible answers gets the most votes among \(k\) independent samples, breaking ties at random.
Simulate this voting scheme for \(p = 0.57\) (the chapter’s post-CoT GSM8K accuracy) with \(m=3\) wrong answers, for \(k \in \{1, 3, 5, 7, 11, 21\}\) samples per problem, using at least 20,000 simulated problems per value of \(k\). Report the accuracy at each \(k\) and the gain of \(k=11\) over the single-sample case \(k=1\).
Consider the simpler two-outcome case: ignore how the wrong answers split, and count a vote as correct whenever more than half of the \(k\) samples are correct. For odd \(k\) this majority-vote probability is a binomial tail, \[ \prob{\text{majority correct}} = \prob{X > k/2}, \qquad X \sim \text{Binomial}(k,p), \quad \E{X} = kp. \] Compute this theoretical accuracy for \(p=0.57\) at the same values of \(k\) as in (a). Is it higher or lower than your simulation from (a), and why would splitting the wrong answers across \(m>1\) labels make the vote easier for the correct answer to win than a strict two-outcome majority requires?
Repeat (a) and (b) with \(p = 0.18\), the chapter’s pre-CoT (direct-answer) GSM8K accuracy. What happens to voting accuracy as \(k\) grows? State a general rule for when self-consistency helps versus when it hurts, and explain why the accuracy gains in (a) shrink at larger \(k\) even though \(p > 0.5\) throughout.
Exercise 24.15 (Tree-of-Thought and Reflexion: Search and Self-Critique) Section 24.4 presents two ways to spend extra inference-time compute beyond the self-consistency voting of Exercise 24.14: Tree-of-Thought (ToT) explores and prunes multiple partial solutions, while Reflexion revises a single line of attempts using its own written critique of what went wrong.
- Tree-of-Thought as pruned beam search. Model a \(d\)-step puzzle (like Game of 24’s three arithmetic operations) where at each step there are \(b = 4\) candidate continuations, exactly one of which lies on a correct solution path. A linear CoT strategy picks one candidate uniformly at random at each of \(d = 3\) steps, succeeding only if it is correct all three times. A ToT strategy scores all \(b\) candidates with a noisy evaluator that ranks the true best candidate first with probability \(s = 0.6\) (otherwise it orders all \(b\) candidates uniformly at random), keeps the best \(k = 2\), and expands both survivors; the search succeeds if the correct candidate survives the beam at every level.
- Derive \(p_{\text{CoT}} = (1/b)^d\) and, with per-level retention probability \(r = s + (1-s)(k/b)\), \(p_{\text{ToT}} = r^d\).
- Compute both.
- Compute the number of nodes each strategy evaluates: linear CoT (\(d\)), beam search (\(b + (d-1)kb\)), and exhaustive search (\(b^d\)). Express the beam search’s savings as a multiple of the exhaustive cost.
- Confirm (ii) with a short Monte Carlo simulation, and comment on how the pattern (large accuracy gain, modest node-count increase) compares to the chapter’s Game of 24 result (4% to 74%, Yao et al. (2023)).
- Reflexion and the independence assumption. The chapter reports a GPT-4 agent using Reflexion raised its HumanEval success rate from \(p_1 = 0.80\) to 0.91 (Shinn et al. 2023). Suppose, counterfactually, that repeated attempts at the same problem (no reflection, just resampling, checked against the same unit-test oracle) were independent Bernoulli trials each succeeding with probability \(p_1\).
- Compute \(\prob{\text{at least one success in 2 independent tries}}\) and compare it to the reported 0.91.
- Your answer to (i) exceeds 0.91. What does this imply about whether repeated LLM samples on the same failed problem, with no new information, can be treated as independent draws at rate \(p_1\)?
- Using the chapter’s description of Reflexion (“the model gains short-term memory and avoids repeating past mistakes”), argue why the targeted information in a self-written critique does work that plain resampling cannot, and relate this to why the independence assumption in Exercise 24.14 (independent parallel CoT samples) is more defensible than an independence assumption across un-reflected sequential retries on the same problem.
- Fill in a table contrasting self-consistency (Exercise 24.14), Tree-of-Thought, and Reflexion on: whether attempts are generated independently or sequentially/dependently; whether an external verifier or oracle is required; and what happens to a failed candidate.