|
|
| (24 intermediate revisions by 8 users not shown) |
| Line 1: |
Line 1: |
| {{Short description|Training methods for pretrained LLMs}} | | {{Short description|Training methods used after LLM pretraining}} |
| | |
| {{Use American English|date=July 2026}} | | {{Use American English|date=July 2026}} |
|
| |
|
| '''Post-training of large language models''' is the set of training methods applied to a [[Large language model|large language model]] (LLM) after its initial large-scale pretraining. Post-training starts from a pretrained or ''base'' model and changes its behavior for purposes such as following instructions, matching stated preferences, improving multi-step reasoning, adapting to a domain, or transferring capabilities into a smaller or more efficient model.<ref name="Tie2025">{{cite arXiv
| | A [[large language model]] (LLM) needs to be trained. The training of an LLM has become more complex over time. |
| |last1=Tie |first1=Guiyao
| |
| |last2=Zhao |first2=Zeli
| |
| |last3=Song |first3=Dingjie
| |
| |last4=Wei |first4=Fuyang
| |
| |display-authors=3
| |
| |title=A Survey on Post-training of Large Language Models
| |
| |eprint=2503.06072
| |
| |year=2025
| |
| }}</ref><ref name="Kumar2025">{{cite arXiv
| |
| |last1=Kumar |first1=Komal
| |
| |last2=Ashraf |first2=Tajamul
| |
| |last3=Thawakar |first3=Omkar
| |
| |last4=Anwer |first4=Rao Muhammad
| |
| |display-authors=3
| |
| |title=LLM Post-Training: A Deep Dive into Reasoning Large Language Models
| |
| |eprint=2502.21321
| |
| |year=2025
| |
| }}</ref><ref name="Zhao2026">{{cite arXiv
| |
| |last1=Zhao |first1=Shiwan
| |
| |last2=Wang |first2=Zhihu
| |
| |last3=Zhao |first3=Xuyang
| |
| |last4=Zhou |first4=Jiaming
| |
| |display-authors=3
| |
| |title=Large Language Model Post-Training: A Unified View of Off-Policy and On-Policy Learning
| |
| |eprint=2604.07941
| |
| |year=2026
| |
| }}</ref> Common post-training stages include [[Fine-tuning (deep learning)|supervised fine-tuning]], preference optimization, [[Reinforcement learning|reinforcement learning]], process or verifier-guided training, parameter-efficient adaptation, and [[Knowledge distillation|knowledge distillation]].<ref name="Tie2025" /><ref name="Zhao2026" />
| |
| | |
| Post-training commonly uses datasets that are more structured than pretraining corpora and, in many published pipelines, substantially smaller. Depending on the method, the data may contain instruction–response examples, comparisons between candidate responses, critiques generated by people or other models, intermediate reasoning steps, or outcomes that can be checked automatically.<ref name="Zhang2023">{{cite arXiv
| |
| |last1=Zhang |first1=Shengyu
| |
| |last2=Dong |first2=Linfeng
| |
| |last3=Li |first3=Xiaoya
| |
| |last4=Zhang |first4=Sen
| |
| |display-authors=3
| |
| |title=Instruction Tuning for Large Language Models: A Survey
| |
| |eprint=2308.10792
| |
| |year=2023
| |
| }}</ref><ref name="Ouyang2022">{{cite arXiv
| |
| |last1=Ouyang |first1=Long
| |
| |last2=Wu |first2=Jeff
| |
| |last3=Jiang |first3=Xu
| |
| |last4=Almeida |first4=Diogo
| |
| |display-authors=3
| |
| |title=Training language models to follow instructions with human feedback
| |
| |eprint=2203.02155
| |
| |year=2022
| |
| }}</ref><ref name="Uesato2022">{{cite arXiv
| |
| |last1=Uesato |first1=Jonathan
| |
| |last2=Kushman |first2=Nate
| |
| |last3=Kumar |first3=Ramana
| |
| |last4=Song |first4=Francis
| |
| |display-authors=3
| |
| |title=Solving math word problems with process- and outcome-based feedback
| |
| |eprint=2211.14275
| |
| |year=2022
| |
| }}</ref> Modern systems often use several stages in sequence rather than one objective throughout training.<ref name="Lambert2024">{{cite arXiv
| |
| |last1=Lambert |first1=Nathan
| |
| |last2=Morrison |first2=Jacob
| |
| |last3=Pyatkin |first3=Valentina
| |
| |last4=Huang |first4=Shengyi
| |
| |display-authors=3
| |
| |title=Tulu 3: Pushing Frontiers in Open Language Model Post-Training
| |
| |eprint=2411.15124
| |
| |year=2024
| |
| }}</ref><ref name="DeepSeek2025">{{cite arXiv
| |
| |author=DeepSeek-AI
| |
| |title=DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning
| |
| |eprint=2501.12948
| |
| |year=2025
| |
| }}</ref>
| |
| | |
| == Scope and terminology ==
| |
| | |
| The boundary of ''post-training'' is not identical across the literature. Dedicated surveys consistently include supervised fine-tuning, preference-based alignment, reinforcement learning, and reasoning-oriented training, while some broader taxonomies also include efficiency methods, multimodal adaptation, tool integration, and inference-time scaling.<ref name="Tie2025" /><ref name="Kumar2025" /><ref name="Zhao2026" /> Some surveys distinguish weight-updating post-training from inference-time scaling, while broader surveys discuss both within a single account of how pretrained models are adapted.<ref name="Kumar2025" /><ref name="Zhao2026" />
| |
| | |
| The term is also distinct from [[Large language model#Quantization|post-training quantization]], which reduces the numerical precision of an already trained network without referring to the broader process of teaching instruction following, preferences, or reasoning behavior.<ref name="Nagel2020">{{cite conference | |
| |last1=Nagel |first1=Markus
| |
| |last2=Amjad |first2=Rana Ali
| |
| |last3=van Baalen |first3=Mart
| |
| |last4=Louizos |first4=Christos
| |
| |display-authors=3
| |
| |title=Up or Down? Adaptive Rounding for Post-Training Quantization
| |
| |book-title=Proceedings of the 37th International Conference on Machine Learning
| |
| |series=Proceedings of Machine Learning Research
| |
| |volume=119
| |
| |pages=7197–7206
| |
| |year=2020
| |
| |url=https://proceedings.mlr.press/v119/nagel20a.html
| |
| }}</ref><ref name="Tie2025" />
| |
| | |
| Post-training does not necessarily correspond to a single objective. A 2026 survey organizes methods by whether they learn from externally supplied trajectories or from the model's own rollouts, and describes their roles as expanding reachable behavior, reshaping probabilities among reachable behaviors, or consolidating behavior across stages and model sizes.<ref name="Zhao2026" /> This view treats multi-stage recipes as coordinated systems rather than as interchangeable fine-tuning procedures.
| |
| | |
| == Development ==
| |
| | |
| Multitask instruction tuning was among the early methods later grouped under LLM post-training. FLAN fine-tuned a pretrained model on instructions for more than 60 natural-language-processing tasks and improved zero-shot performance on held-out task types.<ref name="Wei2021">{{cite arXiv
| |
| |last1=Wei |first1=Jason
| |
| |last2=Bosma |first2=Maarten
| |
| |last3=Zhao |first3=Vincent Y.
| |
| |last4=Guu |first4=Kelvin
| |
| |display-authors=3
| |
| |title=Finetuned Language Models Are Zero-Shot Learners
| |
| |eprint=2109.01652
| |
| |year=2021
| |
| }}</ref> T0 similarly converted many supervised datasets into prompted tasks and trained a model to generalize to tasks that were excluded from its training mixture.<ref name="Sanh2021">{{cite arXiv
| |
| |last1=Sanh |first1=Victor
| |
| |last2=Webson |first2=Albert
| |
| |last3=Raffel |first3=Colin
| |
| |last4=Bach |first4=Stephen H.
| |
| |display-authors=3
| |
| |title=Multitask Prompted Training Enables Zero-Shot Task Generalization
| |
| |eprint=2110.08207
| |
| |year=2021
| |
| }}</ref> These projects helped establish instruction tuning as a way to bridge the next-token-prediction objective used in pretraining and the user-facing objective of responding to natural-language requests.<ref name="Zhang2023" />
| |
| | |
| Preference-based post-training developed alongside instruction tuning. Work on abstractive summarization trained a reward model from human comparisons and then optimized a language model against that reward.<ref name="Stiennon2020">{{cite arXiv
| |
| |last1=Stiennon |first1=Nisan
| |
| |last2=Ouyang |first2=Long
| |
| |last3=Wu |first3=Jeff
| |
| |last4=Ziegler |first4=Daniel M.
| |
| |display-authors=3
| |
| |title=Learning to summarize from human feedback
| |
| |eprint=2009.01325
| |
| |year=2020
| |
| }}</ref> InstructGPT used a sequence of supervised fine-tuning on demonstrations, reward-model training on ranked outputs, and reinforcement learning from human feedback (RLHF).<ref name="Ouyang2022" /> Later RLHF systems and surveys retained this general sequence while identifying limitations in the collection, aggregation, and optimization of human feedback.<ref name="Casper2023">{{cite journal
| |
| |last1=Casper |first1=Stephen
| |
| |last2=Davies |first2=Xander
| |
| |last3=Shi |first3=Claudia
| |
| |last4=Gilbert |first4=Thomas Krendl
| |
| |display-authors=3
| |
| |title=Open Problems and Fundamental Limitations of Reinforcement Learning from Human Feedback
| |
| |journal=Transactions on Machine Learning Research
| |
| |date=18 September 2023
| |
| |arxiv=2307.15217
| |
| }}</ref>
| |
| | |
| Direct preference optimization (DPO), introduced in 2023, reformulated a commonly used preference-learning objective as a classification loss and removed the need to train a separate reward model and run an online reinforcement-learning loop.<ref name="Rafailov2023">{{cite arXiv
| |
| |last1=Rafailov |first1=Rafael
| |
| |last2=Sharma |first2=Archit
| |
| |last3=Mitchell |first3=Eric
| |
| |last4=Ermon |first4=Stefano
| |
| |display-authors=3
| |
| |title=Direct Preference Optimization: Your Language Model is Secretly a Reward Model
| |
| |eprint=2305.18290
| |
| |year=2023
| |
| }}</ref> Other objectives broadened the forms of feedback that could be used. Kahneman–Tversky optimization (KTO), for example, was designed to learn from a binary signal indicating whether an output was desirable rather than from paired rankings.<ref name="Ethayarajh2024">{{cite arXiv
| |
| |last1=Ethayarajh |first1=Kawin
| |
| |last2=Xu |first2=Winnie
| |
| |last3=Muennighoff |first3=Niklas
| |
| |last4=Jurafsky |first4=Dan
| |
| |display-authors=3
| |
| |title=KTO: Model Alignment as Prospect Theoretic Optimization
| |
| |eprint=2402.01306
| |
| |year=2024
| |
| }}</ref>
| |
| | |
| Post-training recipes increasingly combined several data sources and objectives. The open Tulu 3 recipe used supervised fine-tuning, DPO, and reinforcement learning with automatically verifiable rewards, together with dataset curation, decontamination, and separate development and unseen evaluations.<ref name="Lambert2024" /> Reasoning-focused pipelines also combined generated rationales, process supervision, rule-based or learned verifiers, reinforcement learning, and distillation.<ref name="Uesato2022" /><ref name="Lightman2023">{{cite arXiv
| |
| |last1=Lightman |first1=Hunter
| |
| |last2=Kosaraju |first2=Vineet
| |
| |last3=Burda |first3=Yura
| |
| |last4=Edwards |first4=Harri
| |
| |display-authors=3
| |
| |title=Let's Verify Step by Step
| |
| |eprint=2305.20050
| |
| |year=2023
| |
| }}</ref><ref name="DeepSeek2025" />
| |
| | |
| == Methods ==
| |
| | |
| === Supervised fine-tuning and instruction tuning ===
| |
| | |
| In supervised fine-tuning (SFT), the model is trained to predict a target response for a given input. Instruction tuning is a form of SFT in which examples are organized as natural-language instructions paired with desired outputs. Surveys of instruction tuning examine the construction of instruction datasets, the generation or collection of outputs, training mixtures, and the effects of data quantity and quality.<ref name="Zhang2023" />
| |
| | |
| Instruction data can be written by people, converted from existing task datasets, generated synthetically, or assembled from several sources. Self-Instruct used a language model to generate instructions, inputs, and outputs, filtered invalid or near-duplicate examples, and then used the resulting data to fine-tune the model.<ref name="Wang2022">{{cite arXiv
| |
| |last1=Wang |first1=Yizhong
| |
| |last2=Kordi |first2=Yeganeh
| |
| |last3=Mishra |first3=Swaroop
| |
| |last4=Liu |first4=Alisa
| |
| |display-authors=3
| |
| |title=Self-Instruct: Aligning Language Models with Self-Generated Instructions
| |
| |eprint=2212.10560
| |
| |year=2022
| |
| }}</ref> Experiments with LIMA used 1,000 curated prompt–response pairs and argued that careful example selection can be important when a pretrained model already contains broad knowledge.<ref name="Zhou2023">{{cite arXiv
| |
| |last1=Zhou |first1=Chunting
| |
| |last2=Liu |first2=Pengfei
| |
| |last3=Xu |first3=Puxin
| |
| |last4=Iyer |first4=Srini
| |
| |display-authors=3
| |
| |title=LIMA: Less Is More for Alignment
| |
| |eprint=2305.11206
| |
| |year=2023
| |
| }}</ref> These results do not imply that a fixed amount of instruction data is sufficient for every model or task; instruction-tuning surveys report that outcomes depend on the model, task distribution, data construction, and training procedure.<ref name="Zhang2023" />
| |
| | |
| SFT is often the first stage of a longer post-training pipeline. It can establish response formats and basic instruction following before preference optimization or reinforcement learning, and it can provide a starting policy that generates outputs for later comparison or verification.<ref name="Ouyang2022" /><ref name="Lambert2024" /><ref name="Zhao2026" />
| |
| | |
| === Preference learning and alignment ===
| |
| | |
| Preference learning uses judgments about candidate outputs rather than only a single target answer. Feedback can be collected as rankings, pairwise comparisons, or labels such as desirable and undesirable. The training objective then increases the likelihood of preferred behavior while usually constraining the updated model from moving too far from a reference policy.<ref name="Ouyang2022" /><ref name="Rafailov2023" /><ref name="Ethayarajh2024" />
| |
| | |
| ==== Reward modeling and RLHF ====
| |
| | |
| A common RLHF pipeline begins with SFT, collects comparisons between model outputs, trains a reward model to predict those comparisons, and optimizes the language-model policy to increase the predicted reward.<ref name="Ouyang2022" /><ref name="Stiennon2020" /> The reward model is a learned proxy for the feedback data rather than a direct measurement of a universal human preference. InstructGPT, for example, described alignment relative to the preferences of its labelers and researchers, not to all possible users or value systems.<ref name="Ouyang2022" />
| |
| | |
| Reward models can be evaluated separately from the policies they train. RewardBench assembled prompt–response pairs covering chat, reasoning, and safety cases to compare reward models and DPO-style implicit reward functions.<ref name="RewardBench2024">{{cite arXiv
| |
| |last1=Lambert |first1=Nathan
| |
| |last2=Pyatkin |first2=Valentina
| |
| |last3=Morrison |first3=Jacob
| |
| |last4=Miranda |first4=LJ
| |
| |display-authors=3
| |
| |title=RewardBench: Evaluating Reward Models for Language Modeling
| |
| |eprint=2403.13787
| |
| |year=2024
| |
| }}</ref> Such evaluation is relevant because a policy can exploit errors or blind spots in a learned reward model when optimization becomes stronger.<ref name="Casper2023" /><ref name="Rafailov2024">{{cite arXiv
| |
| |last1=Rafailov |first1=Rafael
| |
| |last2=Chittepu |first2=Yaswanth
| |
| |last3=Park |first3=Ryan
| |
| |last4=Sikchi |first4=Harshit
| |
| |display-authors=3
| |
| |title=Scaling Laws for Reward Model Overoptimization in Direct Alignment Algorithms
| |
| |eprint=2406.02900
| |
| |year=2024
| |
| }}</ref>
| |
| | |
| ==== Direct preference optimization ====
| |
| | |
| DPO learns directly from preferred and rejected responses. Its derivation expresses the optimal policy for a class of reward-regularized objectives in a form that can be fitted with a binary classification loss, avoiding a separately trained reward model and an online policy-gradient stage.<ref name="Rafailov2023" /> DPO nevertheless depends on the coverage and quality of the comparison data and on choices such as the reference policy and regularization strength.<ref name="Rafailov2023" /><ref name="Rafailov2024" />
| |
| | |
| Related direct-alignment objectives use different feedback assumptions. KTO was proposed for settings in which examples are labeled desirable or undesirable without a matched alternative response.<ref name="Ethayarajh2024" /> These objectives are alternatives within preference-based post-training rather than evidence that one loss is uniformly best across datasets and applications.<ref name="Ethayarajh2024" /><ref name="Zhao2026" />
| |
| | |
| ==== AI-generated feedback ====
| |
| | |
| Feedback can also be produced or assisted by another language model. Constitutional AI used a written set of principles to guide model-generated critiques and revisions, followed by preference learning from AI feedback.<ref name="Bai2022">{{cite arXiv
| |
| |last1=Bai |first1=Yuntao
| |
| |last2=Kadavath |first2=Saurav
| |
| |last3=Kundu |first3=Sandipan
| |
| |last4=Askell |first4=Amanda
| |
| |display-authors=3
| |
| |title=Constitutional AI: Harmlessness from AI Feedback
| |
| |eprint=2212.08073
| |
| |year=2022
| |
| }}</ref> Reinforcement learning from AI feedback (RLAIF) trains a reward model using preferences generated by an AI labeler. In experiments on summarization and dialogue tasks, RLAIF produced results comparable to the studied RLHF baselines, while reducing dependence on human preference labels.<ref name="Lee2023">{{cite arXiv
| |
| |last1=Lee |first1=Harrison
| |
| |last2=Phatale |first2=Samrat
| |
| |last3=Mansoor |first3=Hassan
| |
| |last4=Mesnard |first4=Thomas
| |
| |display-authors=3
| |
| |title=RLAIF vs. RLHF: Scaling Reinforcement Learning from Human Feedback with AI Feedback
| |
| |eprint=2309.00267
| |
| |year=2023
| |
| }}</ref>
| |
| | |
| AI-generated feedback reduces dependence on human preference labels, but its results depend on the capabilities and prompts of the labeler model, the written principles used to guide judgments, and separate validation procedures.<ref name="Bai2022" /><ref name="Lee2023" /><ref name="Casper2023" />
| |
| | |
| === Reasoning-focused post-training ===
| |
| | |
| Reasoning-oriented post-training uses examples or rewards that distinguish successful multi-step problem solving from plausible but incorrect output. Training signals can include worked solutions, model-generated rationales that pass a verifier, feedback on intermediate steps, or rewards based on a checkable final answer.<ref name="Kumar2025" /><ref name="Uesato2022" /><ref name="Lightman2023" />
| |
| | |
| The Self-Taught Reasoner (STaR) iteratively generated rationales, retained rationales that led to correct answers, and fine-tuned on the retained examples.<ref name="Zelikman2022">{{cite arXiv
| |
| |last1=Zelikman |first1=Eric
| |
| |last2=Wu |first2=Yuhuai
| |
| |last3=Mu |first3=Jesse
| |
| |last4=Goodman |first4=Noah D.
| |
| |title=STaR: Bootstrapping Reasoning With Reasoning
| |
| |eprint=2203.14465
| |
| |year=2022
| |
| }}</ref> Research comparing process and outcome supervision found that feedback on intermediate reasoning steps can reduce reasoning errors that are not visible from the final answer alone.<ref name="Uesato2022" /> ''Let's Verify Step by Step'' trained process reward models on step-level judgments; in the studied mathematical-reasoning setting, those models outperformed outcome-supervised reward models.<ref name="Lightman2023" />
| |
| | |
| When outcomes can be checked automatically, reinforcement learning can use rule-based or programmatic rewards instead of subjective preference labels. DeepSeekMath introduced group relative policy optimization (GRPO), a variant of proximal policy optimization designed to improve mathematical reasoning while reducing the memory requirements of policy optimization.<ref name="Shao2024">{{cite arXiv
| |
| |last1=Shao |first1=Zhihong
| |
| |last2=Wang |first2=Peiyi
| |
| |last3=Zhu |first3=Qihao
| |
| |last4=Xu |first4=Runxin
| |
| |display-authors=3
| |
| |title=DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models
| |
| |eprint=2402.03300
| |
| |year=2024
| |
| }}</ref> DeepSeek-R1 used a multi-stage pipeline that combined reinforcement learning, supervised fine-tuning on filtered reasoning data, further reinforcement learning, and distillation into smaller models.<ref name="DeepSeek2025" /> These systems illustrate one family of reasoning post-training; their reported results do not establish that the same recipe transfers unchanged to every domain or base model.<ref name="Kumar2025" /><ref name="Zhao2026" />
| |
| | |
| === Parameter-efficient adaptation ===
| |
| | |
| Full fine-tuning updates all model parameters and can require substantial accelerator memory. Parameter-efficient methods instead train a smaller set of added or selected parameters while leaving most base-model weights fixed. Low-rank adaptation (LoRA) represents weight updates with trainable low-rank matrices, reducing the number of trainable parameters and the memory needed to store separate task adaptations.<ref name="Hu2021">{{cite arXiv
| |
| |last1=Hu |first1=Edward J.
| |
| |last2=Shen |first2=Yelong
| |
| |last3=Wallis |first3=Phillip
| |
| |last4=Allen-Zhu |first4=Zeyuan
| |
| |display-authors=3
| |
| |title=LoRA: Low-Rank Adaptation of Large Language Models
| |
| |eprint=2106.09685
| |
| |year=2021
| |
| }}</ref>
| |
| | |
| Quantized low-rank adaptation (QLoRA) combines low-rank adapters with a quantized frozen base model. Its design backpropagates gradients through a frozen 4-bit quantized model into LoRA adapters, reducing memory use during fine-tuning.<ref name="Dettmers2023">{{cite arXiv
| |
| |last1=Dettmers |first1=Tim
| |
| |last2=Pagnoni |first2=Artidoro
| |
| |last3=Holtzman |first3=Ari
| |
| |last4=Zettlemoyer |first4=Luke
| |
| |title=QLoRA: Efficient Finetuning of Quantized LLMs
| |
| |eprint=2305.14314
| |
| |year=2023
| |
| }}</ref> LoRA and QLoRA describe how an update is parameterized; they can be used with instruction-tuning or other post-training objectives rather than constituting a separate behavioral goal.<ref name="Tie2025" />
| |
| | |
| === Distillation and consolidation ===
| |
| | |
| [[Knowledge distillation]] trains a smaller or otherwise different student model to reproduce behavior from a teacher model. In LLM post-training, the transferred signal can be token probabilities, generated responses, ranked outputs, reasoning traces, or preferences.<ref name="Zhao2026" /> MiniLLM used an on-policy distillation objective based on reverse Kullback–Leibler divergence to train smaller generative language models from larger teachers.<ref name="Gu2023">{{cite arXiv
| |
| |last1=Gu |first1=Yuxian
| |
| |last2=Dong |first2=Li
| |
| |last3=Wei |first3=Furu
| |
| |last4=Huang |first4=Minlie
| |
| |title=MiniLLM: On-Policy Distillation of Large Language Models
| |
| |eprint=2306.08543
| |
| |year=2023
| |
| }}</ref> Zephyr combined distilled SFT data with preference data ranked by a teacher model and applied distilled DPO.<ref name="Tunstall2023">{{cite arXiv
| |
| |last1=Tunstall |first1=Lewis
| |
| |last2=Beeching |first2=Edward
| |
| |last3=Lambert |first3=Nathan
| |
| |last4=Rajani |first4=Nazneen
| |
| |display-authors=3
| |
| |title=Zephyr: Direct Distillation of LM Alignment
| |
| |eprint=2310.16944
| |
| |year=2023
| |
| }}</ref>
| |
|
| |
|
| Distillation can serve both deployment efficiency and behavioral consolidation. A teacher's post-trained behavior may be transferred into a smaller model, or outputs from a reasoning-oriented model may become SFT data for another model.<ref name="Zhao2026" /><ref name="DeepSeek2025" /> The resulting student behavior depends on the teacher outputs and on the coverage and filtering of the distillation dataset.<ref name="Gu2023" /><ref name="Tunstall2023" />
| | == Pretraining == |
| | '''Pretraining''' is the training stage where the objective is next-token prediction. |
|
| |
|
| == Data and evaluation ==
| | The pretraining stage can be split into several substages, one after another, as in [[curriculum learning]]. Each substage focuses on a different kind of learning, with later substages enabled by previous substages. All substages of pretraining would still use next-token prediction objective. |
|
| |
|
| Post-training datasets differ by stage. SFT uses target responses; preference optimization uses comparisons or desirability labels; process supervision uses judgments on intermediate steps; and reinforcement learning with verifiable rewards uses outcomes that can be checked by rules, tests, or formal answer matching.<ref name="Zhang2023" /><ref name="Ouyang2022" /><ref name="Uesato2022" /><ref name="Lambert2024" /> Synthetic data can expand coverage, but filtering is needed to remove invalid, repetitive, or low-quality generations.<ref name="Wang2022" /><ref name="Zelikman2022" />
| | === Midtraining === |
| | '''Midtraining''' is a substage in pretraining. Like pretraining, the objective in midtraining is next-token prediction. They mainly differ in the design goals of the dataset that the model is predicting about. The pretraining dataset is designed to be larger and more diverse (such as [[Common Crawl]] or [[The Pile (dataset)|the Pile]]) to make the model generally capable, and the midtraining dataset is designed to be smaller, more specialized, and of higher quality, to efficiently make the model capable in specific areas. |
|
| |
|
| Evaluation is normally matched to the intended effect of each stage. Instruction following can be assessed on held-out tasks and human judgments; reward models can be compared on preference test sets; reasoning systems can be evaluated using answer correctness and, when available, step-level judgments; and safety-oriented post-training requires separate adversarial or policy-relevant tests.<ref name="Wei2021" /><ref name="RewardBench2024" /><ref name="Lightman2023" /><ref name="Qi2023">{{cite arXiv
| | The term "mid-training" was first used by OpenAI in 2024,<ref>{{Cite arXiv |eprint=2506.20512 |class=cs.CL |first1=Zengzhi |last1=Wang |first2=Fan |last2=Zhou |title=OctoThinker: Mid-training Incentivizes Reinforcement Learning Scaling |date=2025-06-25 |last3=Li |first3=Xuefeng |last4=Liu |first4=Pengfei}}</ref> and came into more widespread use in 2025.<ref>{{Cite web |title=What's the deal with mid-training? {{!}} Vintage Data |url=https://vintagedata.org/blog/posts/what-is-mid-training |access-date=2026-10-19 |website=vintagedata.org}}</ref><ref>{{Cite arXiv |eprint=2501.00656 |class=cs.CL |last1=OLMo Team |title=2 OLMo 2 Furious |date=2025-10-08 }}</ref> |
| |last1=Qi |first1=Xiangyu
| |
| |last2=Zeng |first2=Yi
| |
| |last3=Xie |first3=Tinghao
| |
| |last4=Chen |first4=Pin-Yu
| |
| |display-authors=3
| |
| |title=Fine-tuning Aligned Language Models Compromises Safety, Even When Users Do Not Intend To!
| |
| |eprint=2310.03693
| |
| |year=2023
| |
| }}</ref> Multi-stage recipes may use development benchmarks during training and separate unseen evaluations afterward. Tulu 3 also applied benchmark decontamination to reduce overlap between training examples and evaluation data.<ref name="Lambert2024" /> | |
|
| |
|
| No single score measures all post-training goals. Improvements in instruction following, reward-model accuracy, reasoning correctness, diversity, harmlessness, or domain performance can move differently under the same update.<ref name="Kirk2023">{{cite arXiv
| | === Long-context training === |
| |last1=Kirk |first1=Robert
| | {{Main|Context window#Context extension}} |
| |last2=Mediratta |first2=Ishita
| | '''Long-context training''' is a training substage within pretraining, designed to increase the effective context length of a model.<ref>{{Cite journal |last1=Gao |first1=Tianyu |last2=Wettig |first2=Alexander |last3=Yen |first3=Howard |last4=Chen |first4=Danqi |date=July 2025 |editor-last=Che |editor-first=Wanxiang |editor2-last=Nabende |editor2-first=Joyce |editor3-last=Shutova |editor3-first=Ekaterina |editor4-last=Pilehvar |editor4-first=Mohammad Taher |title=How to Train Long-Context Language Models (Effectively) |url=https://aclanthology.org/2025.acl-long.366/ |journal=Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) |location=Vienna, Austria |publisher=Association for Computational Linguistics |pages=7376–7399 |doi=10.18653/v1/2025.acl-long.366 |isbn=979-8-89176-251-0}}</ref> A language model can have a long context length, but unable to use it effectively, a phenomenon sometimes called "context decay". Consequently, it is the ''effective'' context length that is of interest, i.e. the length of the context that a model can use effectively.<ref>{{Cite arXiv |eprint=2404.06654 |class=cs.CL |first1=Cheng-Ping |last1=Hsieh |first2=Simeng |last2=Sun |title=RULER: What's the Real Context Size of Your Long-Context Language Models? |date=2024-08-06 |last3=Kriman |first3=Samuel |last4=Acharya |first4=Shantanu |last5=Rekesh |first5=Dima |last6=Jia |first6=Fei |last7=Zhang |first7=Yang |last8=Ginsburg |first8=Boris}}</ref> |
| |last3=Nalmpantis |first3=Christoforos
| |
| |last4=Luketina |first4=Jelena
| |
| |display-authors=3
| |
| |title=Understanding the Effects of RLHF on LLM Generalisation and Diversity
| |
| |eprint=2310.06452
| |
| |year=2023
| |
| }}</ref><ref name="Casper2023" /> Comparisons are also sensitive to the base model, training data, prompts, decoding settings, and evaluator, so results from one pipeline do not by themselves identify a universally preferable method.<ref name="Zhang2023" /><ref name="Ethayarajh2024" /><ref name="Zhao2026" /> | |
|
| |
|
| == Limitations == | | A simple method of long-context training is simply to do next-token prediction on extended contexts, possibly extending more than once. Concretely, the model can be pretrained on context lengths up to 8K tokens, then up to 64K, then up to 512K. It was found that given a fixed training budget, training on a mixture of long contexts and short contexts resulted in higher performance than training on only long contexts.<ref>{{Cite journal |last1=Gao |first1=Tianyu |last2=Wettig |first2=Alexander |last3=Yen |first3=Howard |last4=Chen |first4=Danqi |date=July 2025 |editor-last=Che |editor-first=Wanxiang |editor2-last=Nabende |editor2-first=Joyce |editor3-last=Shutova |editor3-first=Ekaterina |editor4-last=Pilehvar |editor4-first=Mohammad Taher |title=How to Train Long-Context Language Models (Effectively) |url=https://aclanthology.org/2025.acl-long.366/ |journal=Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) |location=Vienna, Austria |publisher=Association for Computational Linguistics |pages=7376–7399 |doi=10.18653/v1/2025.acl-long.366 |isbn=979-8-89176-251-0}}</ref> |
|
| |
|
| Post-training can improve targeted behavior while degrading other properties. Experiments comparing SFT and RLHF found a trade-off in the studied settings: RLHF improved out-of-distribution generalization relative to SFT but reduced output diversity.<ref name="Kirk2023" /> Fine-tuning can also weaken previously applied safety behavior; red-teaming experiments showed that small fine-tuning datasets, including datasets without an explicitly malicious stated purpose, could produce substantial safety regressions in the tested aligned models.<ref name="Qi2023" />
| | Some architectures were designed specifically to enable efficient long-context training. An early example is [[Transformer (deep learning)#ALiBi|ALiBi (Attention with Linear Biases)]],<ref>{{Cite arXiv |eprint=2108.12409 |class=cs.CL |first1=Ofir |last1=Press |first2=Noah A. |last2=Smith |title=Train Short, Test Long: Attention with Linear Biases Enables Input Length Extrapolation |date=2022-04-22 |last3=Lewis |first3=Mike}}</ref> which had been replaced by various schemes for modifying the [[rotary positional embedding]] (RoPE) of a model. The modern representative is YaRN (Yet another [[Rotary positional embedding|RoPE]] extensioN) which has been used across many models, including [[DeepSeek (AI)|DeepSeek]], [[Qwen]], [[Kimi (AI)|Kimi]], and [[GPT-OSS]].<ref>{{Cite journal |last1=Peng |first1=Bowen |last2=Quesnelle |first2=Jeffrey |last3=Fan |first3=Honglu |last4=Shippole |first4=Enrico |date=2024-05-31 |title=YaRN: Efficient Context Window Extension of Large Language Models |url=https://proceedings.iclr.cc/paper_files/paper/2024/hash/874a4d89f2d04b4bcf9a2c19545cf040-Abstract-Conference.html |journal=International Conference on Learning Representations |language=en |volume=2024 |pages=31932–31951}}</ref> |
|
| |
|
| Preference-based methods depend on imperfect feedback. Human labels can be inconsistent, unrepresentative, or difficult to obtain for tasks whose correctness is hard to judge, while AI-generated labels depend on the labeler model, its prompts, and the criteria used to guide judgments.<ref name="Casper2023" /><ref name="Bai2022" /><ref name="Lee2023" /> A learned reward can be overoptimized: further improvement on the proxy reward may cease to correspond to improvement under a stronger or external evaluator.<ref name="Rafailov2024" /> This problem is related to reward hacking and motivates held-out evaluation, regularization, auditing, and the use of independently checkable outcomes when they are available.<ref name="Casper2023" /><ref name="Lambert2024" />
| | == Supervised fine-tuning == |
| | '''Supervised fine-tuning (SFT)''' is the stage where the objective is next-token prediction only on a subset of tokens. Like pretraining, considerations when designing and evaluating the effectiveness of SFT include data composition, context length, token efficiency, downstream performance, and so on. The data composition during the SFT stage critically affects the efficiency of SFT.<ref>{{Cite journal |last1=Xue |first1=Chao |last2=Wang |first2=Yao |last3=Liu |first3=Mengqiao |last4=Liang |first4=Di |last5=Han |first5=Xingsheng |last6=Liu |first6=Peiyang |last7=Wu |first7=Xianjie |last8=Lu |first8=Chenyao |last9=Jiang |first9=Lei |last10=Lu |first10=Yu |last11=Shi |first11=Haibo |last12=Liang |first12=Shuang |last13=Peng |first13=Minlong |last14=Salim |first14=Flora D. |date=July 2026 |editor-last=Liakata |editor-first=Maria |editor2-last=Moreira |editor2-first=Viviane P. |editor3-last=Zhang |editor3-first=Jiajun |editor4-last=Jurgens |editor4-first=David |title=Why Supervised Fine-Tuning Fails to Learn: A Systematic Study of Incomplete Learning in Large Language Models |url=https://aclanthology.org/2026.acl-long.1393/ |journal=Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) |location=San Diego, California, United States |publisher=Association for Computational Linguistics |pages=30186–30213 |doi=10.18653/v1/2026.acl-long.1393 |isbn=979-8-89176-390-6}}</ref><ref>{{Cite journal |last1=Dong |first1=Guanting |last2=Yuan |first2=Hongyi |last3=Lu |first3=Keming |last4=Li |first4=Chengpeng |last5=Xue |first5=Mingfeng |last6=Liu |first6=Dayiheng |last7=Wang |first7=Wei |last8=Yuan |first8=Zheng |last9=Zhou |first9=Chang |last10=Zhou |first10=Jingren |date=August 2024 |editor-last=Ku |editor-first=Lun-Wei |editor2-last=Martins |editor2-first=Andre |editor3-last=Srikumar |editor3-first=Vivek |title=How Abilities in Large Language Models are Affected by Supervised Fine-tuning Data Composition |url=https://aclanthology.org/2024.acl-long.12/ |journal=Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) |location=Bangkok, Thailand |publisher=Association for Computational Linguistics |pages=177–198 |doi=10.18653/v1/2024.acl-long.12}}</ref> |
|
| |
|
| Sequential training stages can interfere with one another. Surveys identify catastrophic forgetting, capability regressions, and trade-offs between alignment objectives and general benchmark performance as recurring concerns.<ref name="Kumar2025" /><ref name="Tie2025" /> Multi-stage pipelines also increase computational and engineering cost and make attribution difficult: a final behavior may depend on the base model, data curation, the order of stages, reward design, sampling, and evaluation choices.<ref name="Lambert2024" /><ref name="Zhao2026" /> Limited disclosure of post-training datasets and recipes can therefore hinder independent reproduction and comparison.<ref name="Lambert2024" />
| | === Instruction tuning === |
| | '''Instruction tuning''' is a type of SFT, in which the SFT dataset contains passages of the form <code><task instruction> <performing the task></code>. The objective is to predict only the tokens in the <code><performing the task></code> part. |
|
| |
|
| == See also == | | == See also == |
| Line 373: |
Line 42: |
|
| |
|
| {{Reflist}} | | {{Reflist}} |
| | {{Artificial intelligence navbox}} |
|
| |
|
| [[Category:Large language models]] | | [[Category:Large language models]] |
| [[Category:Natural language processing]] | | [[Category:Natural language processing]] |