Archive:Training of large language models

Revision as of 17:27, 12 July 2026 by wikipedia>MargaritaPLV (Reworked references to use final journal and conference publications where available)

Template:Short description Template:Use American English

Post-training of large language models is an umbrella term for training applied to a large language model (LLM) after its initial large-scale pretraining. It starts from a pretrained or base model and changes its behavior for purposes such as following instructions, matching stated preferences, adapting to a domain, improving multi-step reasoning, or transferring behavior to a smaller model.[1][2] Common components include supervised fine-tuning, instruction tuning, preference optimization, reinforcement learning, process or verifier-guided training, parameter-efficient adaptation, and knowledge distillation.[3][4]

Post-training data are usually more structured than pretraining corpora. Depending on the method, examples may contain instructions paired with target responses, comparisons between candidate responses, labels indicating desirable or undesirable output, model-generated critiques, intermediate reasoning steps, or outcomes that can be checked automatically.[5][6] Practical systems often combine several stages rather than optimize one objective throughout training.[7]

Scope and terminology

The boundary of post-training varies across the literature. Recent surveys use it for overlapping sets of methods centered on adapting model behavior after pretraining, but they differ on whether to include topics such as multimodal adaptation, tool use, efficiency methods, and inference-time scaling.[1][2] More established literatures treat instruction tuning and reinforcement learning from human feedback (RLHF) as distinct but frequently combined components of model adaptation.[3][4][5]

The term is distinct from post-training quantization, which reduces the numerical precision of an already trained network without referring to the broader process of teaching instruction following, preferences, or reasoning behavior.[8]

Post-training also does not imply a single objective. A pipeline may first establish basic instruction-following behavior with supervised examples, then train on preference data, and finally use reinforcement learning or distillation for a narrower capability or deployment target.[6][7]

Development

Multitask instruction tuning was an early method later grouped within LLM post-training. T0 converted many supervised datasets into prompted tasks and trained a model to generalize to task types excluded from its training mixture.[9] Later FLAN work scaled instruction tuning across model families, task collections, model sizes, and chain-of-thought data.[5]

Preference-based post-training developed alongside instruction tuning. Work on abstractive summarization trained a reward model from human comparisons and then optimized a language model against that reward.[10] InstructGPT used supervised fine-tuning on demonstrations, reward-model training on ranked outputs, and reinforcement learning from human feedback.[6] Reviews of RLHF describe the approach as combining human-feedback collection, reward learning, and policy optimization, while also identifying limitations in each stage.[4][11]

Direct preference optimization (DPO), introduced in 2023, reformulated a commonly used preference-learning objective as a classification loss, avoiding a separately trained reward model and an online policy-gradient stage.[12] Kahneman-Tversky optimization (KTO) was designed for data labeled desirable or undesirable without requiring a matched pair of responses.[13]

Methods

Supervised fine-tuning and instruction tuning

In supervised fine-tuning (SFT), a model is trained to predict a target response for a given input. Instruction tuning is a form of SFT in which examples are organized as natural-language instructions paired with desired outputs. Research on instruction tuning examines task mixtures, instruction and response construction, model scale, data quality, and generalization to unseen tasks.[3][5]

Instruction data can be written by people, converted from existing task datasets, generated synthetically, or assembled from several sources. Self-Instruct used a language model to generate instructions, inputs, and outputs, filtered invalid or near-duplicate examples, and then used the resulting data for fine-tuning.[14] Results from particular instruction-tuning recipes do not imply that a fixed amount or composition of data is sufficient for every model or task; outcomes depend on the base model, task distribution, data construction, and training procedure.[3][5]

SFT is often the first stage of a longer pipeline. It can establish response formats and basic instruction following before preference optimization or reinforcement learning, and it can provide a starting policy that generates outputs for later comparison or verification.[6][7]

Preference learning and alignment

Preference learning uses judgments about candidate outputs rather than only a single target answer. Feedback can be collected as rankings, pairwise comparisons, or labels such as desirable and undesirable. The training objective then increases the probability of preferred behavior while commonly constraining the updated model from moving too far from a reference policy.[4][6][12]

Reward modeling and RLHF

A common RLHF pipeline begins with SFT, collects comparisons between model outputs, trains a reward model to predict those comparisons, and optimizes the language-model policy to increase the predicted reward.[6][10][4] The reward model is a learned proxy for the collected feedback rather than a direct measurement of a universal human preference. The behavior learned by the policy therefore depends on the annotators, instructions, sampling process, and aggregation method used to create the feedback data.[6][11]

Direct preference optimization

DPO learns directly from preferred and rejected responses. Its derivation expresses the optimal policy for a class of reward-regularized objectives in a form that can be fitted with a binary classification loss, eliminating the separately trained reward model and online reinforcement-learning loop used in a common RLHF pipeline.[12] DPO nevertheless depends on the coverage and quality of comparison data and on choices such as the reference policy and regularization strength.[12][15]

Related direct-alignment objectives use different assumptions about feedback. KTO, for example, was proposed for examples labeled desirable or undesirable without a matched alternative response.[13] These objectives are alternatives within preference-based post-training rather than evidence that one loss is uniformly best across datasets and applications.[13][11]

AI-generated feedback

Feedback can be produced or assisted by another language model. Reinforcement learning from AI feedback (RLAIF) trains a reward model using preferences generated by an AI labeler. In experiments on summarization and dialogue tasks, RLAIF produced results comparable to the studied RLHF baselines.[16] AI-generated feedback reduces dependence on human preference labels, but its results depend on the capabilities and prompts of the labeler model and on the criteria used to guide its judgments.[16][11]

Reasoning-focused post-training

Reasoning-oriented post-training uses examples or rewards that distinguish successful multi-step problem solving from plausible but incorrect output. Training signals can include worked solutions, model-generated rationales that pass a verifier, feedback on intermediate steps, or rewards based on a checkable final answer.[2]

The Self-Taught Reasoner (STaR) iteratively generated rationales, retained rationales that led to correct answers, and fine-tuned on the retained examples.[17] Process supervision instead evaluates intermediate reasoning steps. In one study of mathematical reasoning, process-supervised reward models outperformed outcome-supervised reward models in the tested setting.[18] Such results are specific to the studied tasks and models and do not establish that one reasoning-training recipe is uniformly preferable.[2]

Parameter-efficient adaptation

Full fine-tuning updates all model parameters and can require substantial accelerator memory. Parameter-efficient methods instead train a smaller set of added or selected parameters while leaving most base-model weights fixed. Low-rank adaptation (LoRA) represents weight updates with trainable low-rank matrices, reducing the number of trainable parameters and the storage needed for separate task adaptations.[19]

Quantized low-rank adaptation (QLoRA) combines low-rank adapters with a quantized frozen base model. Its design backpropagates gradients through a frozen 4-bit quantized model into LoRA adapters, reducing memory use during fine-tuning.[20] LoRA and QLoRA describe how an update is parameterized; they can be used with instruction tuning or preference optimization rather than constituting a separate behavioral objective.[19][20]

Distillation and consolidation

Knowledge distillation trains a student model to reproduce selected behavior from a teacher model. In LLM post-training, the transferred signal can include token probabilities, generated responses, or instruction-following behavior. MiniLLM used an on-policy distillation objective based on reverse Kullback-Leibler divergence to train smaller generative language models from larger teachers.[21] Distillation can reduce deployment cost or consolidate behavior learned by a larger post-trained model, but the student remains dependent on the teacher outputs and on the coverage and filtering of the distillation data.[21][1]

Data and evaluation

Post-training datasets differ by stage. SFT uses target responses; preference optimization uses comparisons or desirability labels; process supervision uses judgments on intermediate steps; and some reinforcement-learning settings use outcomes that can be checked by rules, tests, or answer matching.[3][4][18] Synthetic data can expand coverage, but filtering is needed to remove invalid, repetitive, or low-quality generations.[14]

Evaluation is normally matched to the intended effect of each stage. Instruction following can be assessed on held-out tasks and human judgments; preference models can be compared on held-out comparisons; reasoning systems can be evaluated using answer correctness and, where available, judgments on intermediate steps; and safety-oriented post-training requires separate adversarial or policy-relevant tests.[5][4][18][22] Multi-stage pipelines may use development benchmarks during training and separate unseen evaluations afterward; some also apply benchmark decontamination to reduce overlap between training and evaluation data.[7]

No single score measures all post-training goals. Improvements in instruction following, preference fit, reasoning correctness, diversity, harmlessness, or domain performance can move differently under the same update.[11][15] Comparisons are also sensitive to the base model, training data, prompts, decoding settings, and evaluator, so results from one pipeline do not by themselves identify a universally preferable method.[3][4]

Limitations

Post-training can improve targeted behavior while degrading other properties. Fine-tuning can weaken previously applied safety behavior; red-teaming experiments have shown safety regressions after fine-tuning with small adversarial datasets and, to a lesser degree, with some benign datasets in the tested models.[22] Sequential stages can also cause capability regressions or catastrophic forgetting, depending on the data and objectives used.[1][2]

Preference-based methods depend on imperfect feedback. Human labels can be inconsistent, unrepresentative, or difficult to obtain for tasks whose correctness is hard to judge, while AI-generated labels depend on the labeler model and its instructions.[4][11][16] Optimization can also overfit a learned or implicit preference signal: performance measured by the training objective may continue to improve while quality under an external evaluator plateaus or deteriorates.[15]

Multi-stage pipelines increase computational and engineering cost and make causal attribution difficult. A final behavior may depend on the base model, data curation, stage order, reward design, sampling, and evaluation choices.[7][1] Limited disclosure of post-training datasets and recipes can hinder independent reproduction and comparison.[7]

See also

References

  1. ↑ 1.0 1.1 1.2 1.3 1.4 Template:Cite arXiv
  2. ↑ 2.0 2.1 2.2 2.3 2.4 Template:Cite arXiv
  3. ↑ 3.0 3.1 3.2 3.3 3.4 3.5 Template:Cite arXiv
  4. ↑ 4.0 4.1 4.2 4.3 4.4 4.5 4.6 4.7 4.8 Kaufmann, Timo; Weng, Paul; Bengs, Viktor; Hüllermeier, Eyke (2025). "A Survey of Reinforcement Learning from Human Feedback". Transactions on Machine Learning Research. 
  5. ↑ 5.0 5.1 5.2 5.3 5.4 5.5 Chung, Hyung Won; Hou, Le; Longpre, Shayne; Zoph, Barret (2024). "Scaling Instruction-Finetuned Language Models". Journal of Machine Learning Research 25 (70): 1–53. 
  6. ↑ 6.0 6.1 6.2 6.3 6.4 6.5 6.6 Template:Cite conference
  7. ↑ 7.0 7.1 7.2 7.3 7.4 7.5 Template:Cite arXiv
  8. ↑ Template:Cite conference
  9. ↑ Template:Cite conference
  10. ↑ 10.0 10.1 Template:Cite conference
  11. ↑ 11.0 11.1 11.2 11.3 11.4 11.5 Casper, Stephen; Davies, Xander; Shi, Claudia; Gilbert, Thomas Krendl (18 September 2023). "Open Problems and Fundamental Limitations of Reinforcement Learning from Human Feedback". Transactions on Machine Learning Research. 
  12. ↑ 12.0 12.1 12.2 12.3 Template:Cite conference
  13. ↑ 13.0 13.1 13.2 Template:Cite conference
  14. ↑ 14.0 14.1 Template:Cite conference
  15. ↑ 15.0 15.1 15.2 Template:Cite conference
  16. ↑ 16.0 16.1 16.2 Template:Cite conference
  17. ↑ Template:Cite conference
  18. ↑ 18.0 18.1 18.2 Template:Cite arXiv
  19. ↑ 19.0 19.1 Template:Cite conference
  20. ↑ 20.0 20.1 Template:Cite conference
  21. ↑ 21.0 21.1 Template:Cite conference
  22. ↑ 22.0 22.1 Template:Cite arXiv