Archive:Training of large language models: Difference between revisions

Added arXiv classifications and disabled mainspace categories in userspace
Reworked references to use final journal and conference publications where available
Line 2: Line 2:
{{Use American English|date=July 2026}}
{{Use American English|date=July 2026}}


'''Post-training of large language models''' is the set of training methods applied to a [[Large language model|large language model]] (LLM) after its initial large-scale pretraining. Post-training starts from a pretrained or ''base'' model and changes its behavior for purposes such as following instructions, matching stated preferences, improving multi-step reasoning, adapting to a domain, or transferring capabilities into a smaller or more efficient model.<ref name="Tie2025">{{cite arXiv
'''Post-training of large language models''' is an umbrella term for training applied to a [[Large language model|large language model]] (LLM) after its initial large-scale pretraining. It starts from a pretrained or ''base'' model and changes its behavior for purposes such as following instructions, matching stated preferences, adapting to a domain, improving multi-step reasoning, or transferring behavior to a smaller model.<ref name="Tie2025">{{cite arXiv
  |last1=Tie |first1=Guiyao
  |last1=Tie |first1=Guiyao
  |last2=Zhao |first2=Zeli
  |last2=Zhao |first2=Zeli
Line 22: Line 22:
  |class=cs.CL
  |class=cs.CL
  |year=2025
  |year=2025
}}</ref><ref name="Zhao2026">{{cite arXiv
}}</ref> Common components include [[Fine-tuning (deep learning)|supervised fine-tuning]], instruction tuning, preference optimization, [[Reinforcement learning|reinforcement learning]], process or verifier-guided training, parameter-efficient adaptation, and [[Knowledge distillation|knowledge distillation]].<ref name="Zhang2023">{{cite arXiv
|last1=Zhao |first1=Shiwan
|last2=Wang |first2=Zhihu
|last3=Zhao |first3=Xuyang
|last4=Zhou |first4=Jiaming
|display-authors=3
|title=Large Language Model Post-Training: A Unified View of Off-Policy and On-Policy Learning
|eprint=2604.07941
|class=cs.CL
|year=2026
}}</ref> Common post-training stages include [[Fine-tuning (deep learning)|supervised fine-tuning]], preference optimization, [[Reinforcement learning|reinforcement learning]], process or verifier-guided training, parameter-efficient adaptation, and [[Knowledge distillation|knowledge distillation]].<ref name="Tie2025" /><ref name="Zhao2026" />
 
Post-training commonly uses datasets that are more structured than pretraining corpora and, in many published pipelines, substantially smaller. Depending on the method, the data may contain instruction&ndash;response examples, comparisons between candidate responses, critiques generated by people or other models, intermediate reasoning steps, or outcomes that can be checked automatically.<ref name="Zhang2023">{{cite arXiv
  |last1=Zhang |first1=Shengyu
  |last1=Zhang |first1=Shengyu
  |last2=Dong |first2=Linfeng
  |last2=Dong |first2=Linfeng
Line 44: Line 32:
  |class=cs.CL
  |class=cs.CL
  |year=2023
  |year=2023
}}</ref><ref name="Ouyang2022">{{cite arXiv
}}</ref><ref name="Kaufmann2025">{{cite journal
|last1=Kaufmann |first1=Timo
|last2=Weng |first2=Paul
|last3=Bengs |first3=Viktor
|last4=Hüllermeier |first4=Eyke
|title=A Survey of Reinforcement Learning from Human Feedback
|journal=Transactions on Machine Learning Research
|year=2025
|arxiv=2312.14925
}}</ref>
 
Post-training data are usually more structured than pretraining corpora. Depending on the method, examples may contain instructions paired with target responses, comparisons between candidate responses, labels indicating desirable or undesirable output, model-generated critiques, intermediate reasoning steps, or outcomes that can be checked automatically.<ref name="Chung2024">{{cite journal
|last1=Chung |first1=Hyung Won
|last2=Hou |first2=Le
|last3=Longpre |first3=Shayne
|last4=Zoph |first4=Barret
|display-authors=3
|title=Scaling Instruction-Finetuned Language Models
|journal=Journal of Machine Learning Research
|volume=25
|issue=70
|pages=1&ndash;53
|year=2024
|arxiv=2210.11416
}}</ref><ref name="Ouyang2022">{{cite conference
  |last1=Ouyang |first1=Long
  |last1=Ouyang |first1=Long
  |last2=Wu |first2=Jeff
  |last2=Wu |first2=Jeff
Line 50: Line 62:
  |last4=Almeida |first4=Diogo
  |last4=Almeida |first4=Diogo
  |display-authors=3
  |display-authors=3
  |title=Training language models to follow instructions with human feedback
  |title=Training Language Models to Follow Instructions with Human Feedback
  |eprint=2203.02155
  |book-title=Advances in Neural Information Processing Systems 35
|class=cs.CL
  |year=2022
  |year=2022
}}</ref><ref name="Uesato2022">{{cite arXiv
  |arxiv=2203.02155
|last1=Uesato |first1=Jonathan
}}</ref> Practical systems often combine several stages rather than optimize one objective throughout training.<ref name="Lambert2024">{{cite arXiv
|last2=Kushman |first2=Nate
|last3=Kumar |first3=Ramana
|last4=Song |first4=Francis
|display-authors=3
|title=Solving math word problems with process- and outcome-based feedback
  |eprint=2211.14275
|class=cs.LG
|year=2022
}}</ref> Modern systems often use several stages in sequence rather than one objective throughout training.<ref name="Lambert2024">{{cite arXiv
  |last1=Lambert |first1=Nathan
  |last1=Lambert |first1=Nathan
  |last2=Morrison |first2=Jacob
  |last2=Morrison |first2=Jacob
Line 74: Line 76:
  |class=cs.CL
  |class=cs.CL
  |year=2024
  |year=2024
}}</ref><ref name="DeepSeek2025">{{cite arXiv
|author=DeepSeek-AI
|title=DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning
|eprint=2501.12948
|class=cs.CL
|year=2025
}}</ref>
}}</ref>


== Scope and terminology ==
== Scope and terminology ==


The boundary of ''post-training'' is not identical across the literature. Dedicated surveys consistently include supervised fine-tuning, preference-based alignment, reinforcement learning, and reasoning-oriented training, while some broader taxonomies also include efficiency methods, multimodal adaptation, tool integration, and inference-time scaling.<ref name="Tie2025" /><ref name="Kumar2025" /><ref name="Zhao2026" /> Some surveys distinguish weight-updating post-training from inference-time scaling, while broader surveys discuss both within a single account of how pretrained models are adapted.<ref name="Kumar2025" /><ref name="Zhao2026" />
The boundary of ''post-training'' varies across the literature. Recent surveys use it for overlapping sets of methods centered on adapting model behavior after pretraining, but they differ on whether to include topics such as multimodal adaptation, tool use, efficiency methods, and inference-time scaling.<ref name="Tie2025" /><ref name="Kumar2025" /> More established literatures treat instruction tuning and reinforcement learning from human feedback (RLHF) as distinct but frequently combined components of model adaptation.<ref name="Zhang2023" /><ref name="Kaufmann2025" /><ref name="Chung2024" />


The term is also distinct from [[Large language model#Quantization|post-training quantization]], which reduces the numerical precision of an already trained network without referring to the broader process of teaching instruction following, preferences, or reasoning behavior.<ref name="Nagel2020">{{cite conference
The term is distinct from [[Large language model#Quantization|post-training quantization]], which reduces the numerical precision of an already trained network without referring to the broader process of teaching instruction following, preferences, or reasoning behavior.<ref name="Nagel2020">{{cite conference
  |last1=Nagel |first1=Markus
  |last1=Nagel |first1=Markus
  |last2=Amjad |first2=Rana Ali
  |last2=Amjad |first2=Rana Ali
Line 99: Line 95:
  |year=2020
  |year=2020
  |url=https://proceedings.mlr.press/v119/nagel20a.html
  |url=https://proceedings.mlr.press/v119/nagel20a.html
}}</ref><ref name="Tie2025" />
}}</ref>


Post-training does not necessarily correspond to a single objective. A 2026 survey organizes methods by whether they learn from externally supplied trajectories or from the model's own rollouts, and describes their roles as expanding reachable behavior, reshaping probabilities among reachable behaviors, or consolidating behavior across stages and model sizes.<ref name="Zhao2026" /> This view treats multi-stage recipes as coordinated systems rather than as interchangeable fine-tuning procedures.
Post-training also does not imply a single objective. A pipeline may first establish basic instruction-following behavior with supervised examples, then train on preference data, and finally use reinforcement learning or distillation for a narrower capability or deployment target.<ref name="Ouyang2022" /><ref name="Lambert2024" />


== Development ==
== Development ==


Multitask instruction tuning was among the early methods later grouped under LLM post-training. FLAN fine-tuned a pretrained model on instructions for more than 60 natural-language-processing tasks and improved zero-shot performance on held-out task types.<ref name="Wei2021">{{cite arXiv
Multitask instruction tuning was an early method later grouped within LLM post-training. T0 converted many supervised datasets into prompted tasks and trained a model to generalize to task types excluded from its training mixture.<ref name="Sanh2022">{{cite conference
|last1=Wei |first1=Jason
|last2=Bosma |first2=Maarten
|last3=Zhao |first3=Vincent Y.
|last4=Guu |first4=Kelvin
|display-authors=3
|title=Finetuned Language Models Are Zero-Shot Learners
|eprint=2109.01652
|class=cs.CL
|year=2021
}}</ref> T0 similarly converted many supervised datasets into prompted tasks and trained a model to generalize to tasks that were excluded from its training mixture.<ref name="Sanh2021">{{cite arXiv
  |last1=Sanh |first1=Victor
  |last1=Sanh |first1=Victor
  |last2=Webson |first2=Albert
  |last2=Webson |first2=Albert
Line 122: Line 108:
  |display-authors=3
  |display-authors=3
  |title=Multitask Prompted Training Enables Zero-Shot Task Generalization
  |title=Multitask Prompted Training Enables Zero-Shot Task Generalization
  |eprint=2110.08207
  |book-title=International Conference on Learning Representations
  |class=cs.LG
  |year=2022
  |year=2021
  |arxiv=2110.08207
}}</ref> These projects helped establish instruction tuning as a way to bridge the next-token-prediction objective used in pretraining and the user-facing objective of responding to natural-language requests.<ref name="Zhang2023" />
}}</ref> Later FLAN work scaled instruction tuning across model families, task collections, model sizes, and chain-of-thought data.<ref name="Chung2024" />


Preference-based post-training developed alongside instruction tuning. Work on abstractive summarization trained a reward model from human comparisons and then optimized a language model against that reward.<ref name="Stiennon2020">{{cite arXiv
Preference-based post-training developed alongside instruction tuning. Work on abstractive summarization trained a reward model from human comparisons and then optimized a language model against that reward.<ref name="Stiennon2020">{{cite conference
  |last1=Stiennon |first1=Nisan
  |last1=Stiennon |first1=Nisan
  |last2=Ouyang |first2=Long
  |last2=Ouyang |first2=Long
Line 133: Line 119:
  |last4=Ziegler |first4=Daniel M.
  |last4=Ziegler |first4=Daniel M.
  |display-authors=3
  |display-authors=3
  |title=Learning to summarize from human feedback
  |title=Learning to Summarize from Human Feedback
  |eprint=2009.01325
  |book-title=Advances in Neural Information Processing Systems 33
|class=cs.CL
  |year=2020
  |year=2020
}}</ref> InstructGPT used a sequence of supervised fine-tuning on demonstrations, reward-model training on ranked outputs, and reinforcement learning from human feedback (RLHF).<ref name="Ouyang2022" /> Later RLHF systems and surveys retained this general sequence while identifying limitations in the collection, aggregation, and optimization of human feedback.<ref name="Casper2023">{{cite journal
|arxiv=2009.01325
}}</ref> InstructGPT used supervised fine-tuning on demonstrations, reward-model training on ranked outputs, and reinforcement learning from human feedback.<ref name="Ouyang2022" /> Reviews of RLHF describe the approach as combining human-feedback collection, reward learning, and policy optimization, while also identifying limitations in each stage.<ref name="Kaufmann2025" /><ref name="Casper2023">{{cite journal
  |last1=Casper |first1=Stephen
  |last1=Casper |first1=Stephen
  |last2=Davies |first2=Xander
  |last2=Davies |first2=Xander
Line 149: Line 135:
}}</ref>
}}</ref>


Direct preference optimization (DPO), introduced in 2023, reformulated a commonly used preference-learning objective as a classification loss and removed the need to train a separate reward model and run an online reinforcement-learning loop.<ref name="Rafailov2023">{{cite arXiv
Direct preference optimization (DPO), introduced in 2023, reformulated a commonly used preference-learning objective as a classification loss, avoiding a separately trained reward model and an online policy-gradient stage.<ref name="Rafailov2023">{{cite conference
  |last1=Rafailov |first1=Rafael
  |last1=Rafailov |first1=Rafael
  |last2=Sharma |first2=Archit
  |last2=Sharma |first2=Archit
Line 155: Line 141:
  |last4=Ermon |first4=Stefano
  |last4=Ermon |first4=Stefano
  |display-authors=3
  |display-authors=3
  |title=Direct Preference Optimization: Your Language Model is Secretly a Reward Model
  |title=Direct Preference Optimization: Your Language Model Is Secretly a Reward Model
  |eprint=2305.18290
  |book-title=Advances in Neural Information Processing Systems 36
|class=cs.LG
  |year=2023
  |year=2023
}}</ref> Other objectives broadened the forms of feedback that could be used. Kahneman&ndash;Tversky optimization (KTO), for example, was designed to learn from a binary signal indicating whether an output was desirable rather than from paired rankings.<ref name="Ethayarajh2024">{{cite arXiv
|arxiv=2305.18290
}}</ref> Kahneman-Tversky optimization (KTO) was designed for data labeled desirable or undesirable without requiring a matched pair of responses.<ref name="Ethayarajh2024">{{cite conference
  |last1=Ethayarajh |first1=Kawin
  |last1=Ethayarajh |first1=Kawin
  |last2=Xu |first2=Winnie
  |last2=Xu |first2=Winnie
Line 166: Line 152:
  |display-authors=3
  |display-authors=3
  |title=KTO: Model Alignment as Prospect Theoretic Optimization
  |title=KTO: Model Alignment as Prospect Theoretic Optimization
  |eprint=2402.01306
  |book-title=Proceedings of the 41st International Conference on Machine Learning
  |class=cs.LG
  |series=Proceedings of Machine Learning Research
|volume=235
  |year=2024
  |year=2024
|arxiv=2402.01306
}}</ref>
}}</ref>
Post-training recipes increasingly combined several data sources and objectives. The open Tulu 3 recipe used supervised fine-tuning, DPO, and reinforcement learning with automatically verifiable rewards, together with dataset curation, decontamination, and separate development and unseen evaluations.<ref name="Lambert2024" /> Reasoning-focused pipelines also combined generated rationales, process supervision, rule-based or learned verifiers, reinforcement learning, and distillation.<ref name="Uesato2022" /><ref name="Lightman2023">{{cite arXiv
|last1=Lightman |first1=Hunter
|last2=Kosaraju |first2=Vineet
|last3=Burda |first3=Yura
|last4=Edwards |first4=Harri
|display-authors=3
|title=Let's Verify Step by Step
|eprint=2305.20050
|class=cs.LG
|year=2023
}}</ref><ref name="DeepSeek2025" />


== Methods ==
== Methods ==
Line 187: Line 163:
=== Supervised fine-tuning and instruction tuning ===
=== Supervised fine-tuning and instruction tuning ===


In supervised fine-tuning (SFT), the model is trained to predict a target response for a given input. Instruction tuning is a form of SFT in which examples are organized as natural-language instructions paired with desired outputs. Surveys of instruction tuning examine the construction of instruction datasets, the generation or collection of outputs, training mixtures, and the effects of data quantity and quality.<ref name="Zhang2023" />
In supervised fine-tuning (SFT), a model is trained to predict a target response for a given input. Instruction tuning is a form of SFT in which examples are organized as natural-language instructions paired with desired outputs. Research on instruction tuning examines task mixtures, instruction and response construction, model scale, data quality, and generalization to unseen tasks.<ref name="Zhang2023" /><ref name="Chung2024" />


Instruction data can be written by people, converted from existing task datasets, generated synthetically, or assembled from several sources. Self-Instruct used a language model to generate instructions, inputs, and outputs, filtered invalid or near-duplicate examples, and then used the resulting data to fine-tune the model.<ref name="Wang2022">{{cite arXiv
Instruction data can be written by people, converted from existing task datasets, generated synthetically, or assembled from several sources. Self-Instruct used a language model to generate instructions, inputs, and outputs, filtered invalid or near-duplicate examples, and then used the resulting data for fine-tuning.<ref name="Wang2023">{{cite conference
  |last1=Wang |first1=Yizhong
  |last1=Wang |first1=Yizhong
  |last2=Kordi |first2=Yeganeh
  |last2=Kordi |first2=Yeganeh
Line 196: Line 172:
  |display-authors=3
  |display-authors=3
  |title=Self-Instruct: Aligning Language Models with Self-Generated Instructions
  |title=Self-Instruct: Aligning Language Models with Self-Generated Instructions
  |eprint=2212.10560
  |book-title=Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics
|class=cs.CL
|year=2022
}}</ref> Experiments with LIMA used 1,000 curated prompt&ndash;response pairs and argued that careful example selection can be important when a pretrained model already contains broad knowledge.<ref name="Zhou2023">{{cite arXiv
|last1=Zhou |first1=Chunting
|last2=Liu |first2=Pengfei
|last3=Xu |first3=Puxin
|last4=Iyer |first4=Srini
|display-authors=3
|title=LIMA: Less Is More for Alignment
|eprint=2305.11206
|class=cs.CL
  |year=2023
  |year=2023
}}</ref> These results do not imply that a fixed amount of instruction data is sufficient for every model or task; instruction-tuning surveys report that outcomes depend on the model, task distribution, data construction, and training procedure.<ref name="Zhang2023" />
|arxiv=2212.10560
}}</ref> Results from particular instruction-tuning recipes do not imply that a fixed amount or composition of data is sufficient for every model or task; outcomes depend on the base model, task distribution, data construction, and training procedure.<ref name="Zhang2023" /><ref name="Chung2024" />


SFT is often the first stage of a longer post-training pipeline. It can establish response formats and basic instruction following before preference optimization or reinforcement learning, and it can provide a starting policy that generates outputs for later comparison or verification.<ref name="Ouyang2022" /><ref name="Lambert2024" /><ref name="Zhao2026" />
SFT is often the first stage of a longer pipeline. It can establish response formats and basic instruction following before preference optimization or reinforcement learning, and it can provide a starting policy that generates outputs for later comparison or verification.<ref name="Ouyang2022" /><ref name="Lambert2024" />


=== Preference learning and alignment ===
=== Preference learning and alignment ===


Preference learning uses judgments about candidate outputs rather than only a single target answer. Feedback can be collected as rankings, pairwise comparisons, or labels such as desirable and undesirable. The training objective then increases the likelihood of preferred behavior while usually constraining the updated model from moving too far from a reference policy.<ref name="Ouyang2022" /><ref name="Rafailov2023" /><ref name="Ethayarajh2024" />
Preference learning uses judgments about candidate outputs rather than only a single target answer. Feedback can be collected as rankings, pairwise comparisons, or labels such as desirable and undesirable. The training objective then increases the probability of preferred behavior while commonly constraining the updated model from moving too far from a reference policy.<ref name="Kaufmann2025" /><ref name="Ouyang2022" /><ref name="Rafailov2023" />


==== Reward modeling and RLHF ====
==== Reward modeling and RLHF ====


A common RLHF pipeline begins with SFT, collects comparisons between model outputs, trains a reward model to predict those comparisons, and optimizes the language-model policy to increase the predicted reward.<ref name="Ouyang2022" /><ref name="Stiennon2020" /> The reward model is a learned proxy for the feedback data rather than a direct measurement of a universal human preference. InstructGPT, for example, described alignment relative to the preferences of its labelers and researchers, not to all possible users or value systems.<ref name="Ouyang2022" />
A common RLHF pipeline begins with SFT, collects comparisons between model outputs, trains a reward model to predict those comparisons, and optimizes the language-model policy to increase the predicted reward.<ref name="Ouyang2022" /><ref name="Stiennon2020" /><ref name="Kaufmann2025" /> The reward model is a learned proxy for the collected feedback rather than a direct measurement of a universal human preference. The behavior learned by the policy therefore depends on the annotators, instructions, sampling process, and aggregation method used to create the feedback data.<ref name="Ouyang2022" /><ref name="Casper2023" />


Reward models can be evaluated separately from the policies they train. RewardBench assembled prompt&ndash;response pairs covering chat, reasoning, and safety cases to compare reward models and DPO-style implicit reward functions.<ref name="RewardBench2024">{{cite arXiv
==== Direct preference optimization ====
|last1=Lambert |first1=Nathan
 
|last2=Pyatkin |first2=Valentina
DPO learns directly from preferred and rejected responses. Its derivation expresses the optimal policy for a class of reward-regularized objectives in a form that can be fitted with a binary classification loss, eliminating the separately trained reward model and online reinforcement-learning loop used in a common RLHF pipeline.<ref name="Rafailov2023" /> DPO nevertheless depends on the coverage and quality of comparison data and on choices such as the reference policy and regularization strength.<ref name="Rafailov2023" /><ref name="Rafailov2024">{{cite conference
|last3=Morrison |first3=Jacob
|last4=Miranda |first4=LJ
|display-authors=3
|title=RewardBench: Evaluating Reward Models for Language Modeling
|eprint=2403.13787
|class=cs.LG
|year=2024
}}</ref> Such evaluation is relevant because a policy can exploit errors or blind spots in a learned reward model when optimization becomes stronger.<ref name="Casper2023" /><ref name="Rafailov2024">{{cite arXiv
  |last1=Rafailov |first1=Rafael
  |last1=Rafailov |first1=Rafael
  |last2=Chittepu |first2=Yaswanth
  |last2=Chittepu |first2=Yaswanth
Line 238: Line 196:
  |display-authors=3
  |display-authors=3
  |title=Scaling Laws for Reward Model Overoptimization in Direct Alignment Algorithms
  |title=Scaling Laws for Reward Model Overoptimization in Direct Alignment Algorithms
  |eprint=2406.02900
  |book-title=Advances in Neural Information Processing Systems 37
|class=cs.LG
  |year=2024
  |year=2024
|arxiv=2406.02900
}}</ref>
}}</ref>


==== Direct preference optimization ====
Related direct-alignment objectives use different assumptions about feedback. KTO, for example, was proposed for examples labeled desirable or undesirable without a matched alternative response.<ref name="Ethayarajh2024" /> These objectives are alternatives within preference-based post-training rather than evidence that one loss is uniformly best across datasets and applications.<ref name="Ethayarajh2024" /><ref name="Casper2023" />
 
DPO learns directly from preferred and rejected responses. Its derivation expresses the optimal policy for a class of reward-regularized objectives in a form that can be fitted with a binary classification loss, avoiding a separately trained reward model and an online policy-gradient stage.<ref name="Rafailov2023" /> DPO nevertheless depends on the coverage and quality of the comparison data and on choices such as the reference policy and regularization strength.<ref name="Rafailov2023" /><ref name="Rafailov2024" />
 
Related direct-alignment objectives use different feedback assumptions. KTO was proposed for settings in which examples are labeled desirable or undesirable without a matched alternative response.<ref name="Ethayarajh2024" /> These objectives are alternatives within preference-based post-training rather than evidence that one loss is uniformly best across datasets and applications.<ref name="Ethayarajh2024" /><ref name="Zhao2026" />


==== AI-generated feedback ====
==== AI-generated feedback ====


Feedback can also be produced or assisted by another language model. Constitutional AI used a written set of principles to guide model-generated critiques and revisions, followed by preference learning from AI feedback.<ref name="Bai2022">{{cite arXiv
Feedback can be produced or assisted by another language model. Reinforcement learning from AI feedback (RLAIF) trains a reward model using preferences generated by an AI labeler. In experiments on summarization and dialogue tasks, RLAIF produced results comparable to the studied RLHF baselines.<ref name="Lee2024">{{cite conference
|last1=Bai |first1=Yuntao
|last2=Kadavath |first2=Saurav
|last3=Kundu |first3=Sandipan
|last4=Askell |first4=Amanda
|display-authors=3
|title=Constitutional AI: Harmlessness from AI Feedback
|eprint=2212.08073
|class=cs.CL
|year=2022
}}</ref> Reinforcement learning from AI feedback (RLAIF) trains a reward model using preferences generated by an AI labeler. In experiments on summarization and dialogue tasks, RLAIF produced results comparable to the studied RLHF baselines, while reducing dependence on human preference labels.<ref name="Lee2023">{{cite arXiv
  |last1=Lee |first1=Harrison
  |last1=Lee |first1=Harrison
  |last2=Phatale |first2=Samrat
  |last2=Phatale |first2=Samrat
Line 268: Line 212:
  |display-authors=3
  |display-authors=3
  |title=RLAIF vs. RLHF: Scaling Reinforcement Learning from Human Feedback with AI Feedback
  |title=RLAIF vs. RLHF: Scaling Reinforcement Learning from Human Feedback with AI Feedback
  |eprint=2309.00267
  |book-title=Proceedings of the 41st International Conference on Machine Learning
  |class=cs.CL
|series=Proceedings of Machine Learning Research
  |year=2023
|volume=235
}}</ref>
  |pages=26874&ndash;26901
 
  |year=2024
AI-generated feedback reduces dependence on human preference labels, but its results depend on the capabilities and prompts of the labeler model, the written principles used to guide judgments, and separate validation procedures.<ref name="Bai2022" /><ref name="Lee2023" /><ref name="Casper2023" />
|arxiv=2309.00267
}}</ref> AI-generated feedback reduces dependence on human preference labels, but its results depend on the capabilities and prompts of the labeler model and on the criteria used to guide its judgments.<ref name="Lee2024" /><ref name="Casper2023" />


=== Reasoning-focused post-training ===
=== Reasoning-focused post-training ===


Reasoning-oriented post-training uses examples or rewards that distinguish successful multi-step problem solving from plausible but incorrect output. Training signals can include worked solutions, model-generated rationales that pass a verifier, feedback on intermediate steps, or rewards based on a checkable final answer.<ref name="Kumar2025" /><ref name="Uesato2022" /><ref name="Lightman2023" />
Reasoning-oriented post-training uses examples or rewards that distinguish successful multi-step problem solving from plausible but incorrect output. Training signals can include worked solutions, model-generated rationales that pass a verifier, feedback on intermediate steps, or rewards based on a checkable final answer.<ref name="Kumar2025" />


The Self-Taught Reasoner (STaR) iteratively generated rationales, retained rationales that led to correct answers, and fine-tuned on the retained examples.<ref name="Zelikman2022">{{cite arXiv
The Self-Taught Reasoner (STaR) iteratively generated rationales, retained rationales that led to correct answers, and fine-tuned on the retained examples.<ref name="Zelikman2022">{{cite conference
  |last1=Zelikman |first1=Eric
  |last1=Zelikman |first1=Eric
  |last2=Wu |first2=Yuhuai
  |last2=Wu |first2=Yuhuai
  |last3=Mu |first3=Jesse
  |last3=Mu |first3=Jesse
  |last4=Goodman |first4=Noah D.
  |last4=Goodman |first4=Noah D.
  |title=STaR: Bootstrapping Reasoning With Reasoning
  |title=STaR: Bootstrapping Reasoning with Reasoning
  |eprint=2203.14465
  |book-title=Advances in Neural Information Processing Systems 35
|class=cs.LG
  |year=2022
  |year=2022
}}</ref> Research comparing process and outcome supervision found that feedback on intermediate reasoning steps can reduce reasoning errors that are not visible from the final answer alone.<ref name="Uesato2022" /> ''Let's Verify Step by Step'' trained process reward models on step-level judgments; in the studied mathematical-reasoning setting, those models outperformed outcome-supervised reward models.<ref name="Lightman2023" />
|arxiv=2203.14465
 
}}</ref> Process supervision instead evaluates intermediate reasoning steps. In one study of mathematical reasoning, process-supervised reward models outperformed outcome-supervised reward models in the tested setting.<ref name="Lightman2023">{{cite arXiv
When outcomes can be checked automatically, reinforcement learning can use rule-based or programmatic rewards instead of subjective preference labels. DeepSeekMath introduced group relative policy optimization (GRPO), a variant of proximal policy optimization designed to improve mathematical reasoning while reducing the memory requirements of policy optimization.<ref name="Shao2024">{{cite arXiv
  |last1=Lightman |first1=Hunter
  |last1=Shao |first1=Zhihong
  |last2=Kosaraju |first2=Vineet
  |last2=Wang |first2=Peiyi
  |last3=Burda |first3=Yura
  |last3=Zhu |first3=Qihao
  |last4=Edwards |first4=Harri
  |last4=Xu |first4=Runxin
  |display-authors=3
  |display-authors=3
  |title=DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models
  |title=Let's Verify Step by Step
  |eprint=2402.03300
  |eprint=2305.20050
  |class=cs.CL
  |class=cs.LG
  |year=2024
  |year=2023
}}</ref> DeepSeek-R1 used a multi-stage pipeline that combined reinforcement learning, supervised fine-tuning on filtered reasoning data, further reinforcement learning, and distillation into smaller models.<ref name="DeepSeek2025" /> These systems illustrate one family of reasoning post-training; their reported results do not establish that the same recipe transfers unchanged to every domain or base model.<ref name="Kumar2025" /><ref name="Zhao2026" />
}}</ref> Such results are specific to the studied tasks and models and do not establish that one reasoning-training recipe is uniformly preferable.<ref name="Kumar2025" />


=== Parameter-efficient adaptation ===
=== Parameter-efficient adaptation ===


Full fine-tuning updates all model parameters and can require substantial accelerator memory. Parameter-efficient methods instead train a smaller set of added or selected parameters while leaving most base-model weights fixed. Low-rank adaptation (LoRA) represents weight updates with trainable low-rank matrices, reducing the number of trainable parameters and the memory needed to store separate task adaptations.<ref name="Hu2021">{{cite arXiv
Full fine-tuning updates all model parameters and can require substantial accelerator memory. Parameter-efficient methods instead train a smaller set of added or selected parameters while leaving most base-model weights fixed. Low-rank adaptation (LoRA) represents weight updates with trainable low-rank matrices, reducing the number of trainable parameters and the storage needed for separate task adaptations.<ref name="Hu2022">{{cite conference
  |last1=Hu |first1=Edward J.
  |last1=Hu |first1=Edward J.
  |last2=Shen |first2=Yelong
  |last2=Shen |first2=Yelong
Line 311: Line 254:
  |display-authors=3
  |display-authors=3
  |title=LoRA: Low-Rank Adaptation of Large Language Models
  |title=LoRA: Low-Rank Adaptation of Large Language Models
  |eprint=2106.09685
  |book-title=International Conference on Learning Representations
  |class=cs.CL
  |year=2022
  |year=2021
  |arxiv=2106.09685
}}</ref>
}}</ref>


Quantized low-rank adaptation (QLoRA) combines low-rank adapters with a quantized frozen base model. Its design backpropagates gradients through a frozen 4-bit quantized model into LoRA adapters, reducing memory use during fine-tuning.<ref name="Dettmers2023">{{cite arXiv
Quantized low-rank adaptation (QLoRA) combines low-rank adapters with a quantized frozen base model. Its design backpropagates gradients through a frozen 4-bit quantized model into LoRA adapters, reducing memory use during fine-tuning.<ref name="Dettmers2023">{{cite conference
  |last1=Dettmers |first1=Tim
  |last1=Dettmers |first1=Tim
  |last2=Pagnoni |first2=Artidoro
  |last2=Pagnoni |first2=Artidoro
Line 322: Line 265:
  |last4=Zettlemoyer |first4=Luke
  |last4=Zettlemoyer |first4=Luke
  |title=QLoRA: Efficient Finetuning of Quantized LLMs
  |title=QLoRA: Efficient Finetuning of Quantized LLMs
  |eprint=2305.14314
  |book-title=Advances in Neural Information Processing Systems 36
|class=cs.LG
  |year=2023
  |year=2023
}}</ref> LoRA and QLoRA describe how an update is parameterized; they can be used with instruction-tuning or other post-training objectives rather than constituting a separate behavioral goal.<ref name="Tie2025" />
|arxiv=2305.14314
}}</ref> LoRA and QLoRA describe how an update is parameterized; they can be used with instruction tuning or preference optimization rather than constituting a separate behavioral objective.<ref name="Hu2022" /><ref name="Dettmers2023" />


=== Distillation and consolidation ===
=== Distillation and consolidation ===


[[Knowledge distillation]] trains a smaller or otherwise different student model to reproduce behavior from a teacher model. In LLM post-training, the transferred signal can be token probabilities, generated responses, ranked outputs, reasoning traces, or preferences.<ref name="Zhao2026" /> MiniLLM used an on-policy distillation objective based on reverse Kullback&ndash;Leibler divergence to train smaller generative language models from larger teachers.<ref name="Gu2023">{{cite arXiv
[[Knowledge distillation]] trains a student model to reproduce selected behavior from a teacher model. In LLM post-training, the transferred signal can include token probabilities, generated responses, or instruction-following behavior. MiniLLM used an on-policy distillation objective based on reverse Kullback-Leibler divergence to train smaller generative language models from larger teachers.<ref name="Gu2024">{{cite conference
  |last1=Gu |first1=Yuxian
  |last1=Gu |first1=Yuxian
  |last2=Dong |first2=Li
  |last2=Dong |first2=Li
Line 335: Line 278:
  |last4=Huang |first4=Minlie
  |last4=Huang |first4=Minlie
  |title=MiniLLM: On-Policy Distillation of Large Language Models
  |title=MiniLLM: On-Policy Distillation of Large Language Models
  |eprint=2306.08543
  |book-title=International Conference on Learning Representations
|class=cs.CL
  |year=2024
|year=2023
  |arxiv=2306.08543
}}</ref> Zephyr combined distilled SFT data with preference data ranked by a teacher model and applied distilled DPO.<ref name="Tunstall2023">{{cite arXiv
}}</ref> Distillation can reduce deployment cost or consolidate behavior learned by a larger post-trained model, but the student remains dependent on the teacher outputs and on the coverage and filtering of the distillation data.<ref name="Gu2024" /><ref name="Tie2025" />
|last1=Tunstall |first1=Lewis
|last2=Beeching |first2=Edward
|last3=Lambert |first3=Nathan
|last4=Rajani |first4=Nazneen
|display-authors=3
|title=Zephyr: Direct Distillation of LM Alignment
  |eprint=2310.16944
  |class=cs.LG
|year=2023
}}</ref>
 
Distillation can serve both deployment efficiency and behavioral consolidation. A teacher's post-trained behavior may be transferred into a smaller model, or outputs from a reasoning-oriented model may become SFT data for another model.<ref name="Zhao2026" /><ref name="DeepSeek2025" /> The resulting student behavior depends on the teacher outputs and on the coverage and filtering of the distillation dataset.<ref name="Gu2023" /><ref name="Tunstall2023" />


== Data and evaluation ==
== Data and evaluation ==


Post-training datasets differ by stage. SFT uses target responses; preference optimization uses comparisons or desirability labels; process supervision uses judgments on intermediate steps; and reinforcement learning with verifiable rewards uses outcomes that can be checked by rules, tests, or formal answer matching.<ref name="Zhang2023" /><ref name="Ouyang2022" /><ref name="Uesato2022" /><ref name="Lambert2024" /> Synthetic data can expand coverage, but filtering is needed to remove invalid, repetitive, or low-quality generations.<ref name="Wang2022" /><ref name="Zelikman2022" />
Post-training datasets differ by stage. SFT uses target responses; preference optimization uses comparisons or desirability labels; process supervision uses judgments on intermediate steps; and some reinforcement-learning settings use outcomes that can be checked by rules, tests, or answer matching.<ref name="Zhang2023" /><ref name="Kaufmann2025" /><ref name="Lightman2023" /> Synthetic data can expand coverage, but filtering is needed to remove invalid, repetitive, or low-quality generations.<ref name="Wang2023" />


Evaluation is normally matched to the intended effect of each stage. Instruction following can be assessed on held-out tasks and human judgments; reward models can be compared on preference test sets; reasoning systems can be evaluated using answer correctness and, when available, step-level judgments; and safety-oriented post-training requires separate adversarial or policy-relevant tests.<ref name="Wei2021" /><ref name="RewardBench2024" /><ref name="Lightman2023" /><ref name="Qi2023">{{cite arXiv
Evaluation is normally matched to the intended effect of each stage. Instruction following can be assessed on held-out tasks and human judgments; preference models can be compared on held-out comparisons; reasoning systems can be evaluated using answer correctness and, where available, judgments on intermediate steps; and safety-oriented post-training requires separate adversarial or policy-relevant tests.<ref name="Chung2024" /><ref name="Kaufmann2025" /><ref name="Lightman2023" /><ref name="Qi2023">{{cite arXiv
  |last1=Qi |first1=Xiangyu
  |last1=Qi |first1=Xiangyu
  |last2=Zeng |first2=Yi
  |last2=Zeng |first2=Yi
Line 366: Line 297:
  |class=cs.CL
  |class=cs.CL
  |year=2023
  |year=2023
}}</ref> Multi-stage recipes may use development benchmarks during training and separate unseen evaluations afterward. Tulu 3 also applied benchmark decontamination to reduce overlap between training examples and evaluation data.<ref name="Lambert2024" />
}}</ref> Multi-stage pipelines may use development benchmarks during training and separate unseen evaluations afterward; some also apply benchmark decontamination to reduce overlap between training and evaluation data.<ref name="Lambert2024" />


No single score measures all post-training goals. Improvements in instruction following, reward-model accuracy, reasoning correctness, diversity, harmlessness, or domain performance can move differently under the same update.<ref name="Kirk2023">{{cite arXiv
No single score measures all post-training goals. Improvements in instruction following, preference fit, reasoning correctness, diversity, harmlessness, or domain performance can move differently under the same update.<ref name="Casper2023" /><ref name="Rafailov2024" /> Comparisons are also sensitive to the base model, training data, prompts, decoding settings, and evaluator, so results from one pipeline do not by themselves identify a universally preferable method.<ref name="Zhang2023" /><ref name="Kaufmann2025" />
|last1=Kirk |first1=Robert
|last2=Mediratta |first2=Ishita
|last3=Nalmpantis |first3=Christoforos
|last4=Luketina |first4=Jelena
|display-authors=3
|title=Understanding the Effects of RLHF on LLM Generalisation and Diversity
|eprint=2310.06452
|class=cs.LG
|year=2023
}}</ref><ref name="Casper2023" /> Comparisons are also sensitive to the base model, training data, prompts, decoding settings, and evaluator, so results from one pipeline do not by themselves identify a universally preferable method.<ref name="Zhang2023" /><ref name="Ethayarajh2024" /><ref name="Zhao2026" />


== Limitations ==
== Limitations ==


Post-training can improve targeted behavior while degrading other properties. Experiments comparing SFT and RLHF found a trade-off in the studied settings: RLHF improved out-of-distribution generalization relative to SFT but reduced output diversity.<ref name="Kirk2023" /> Fine-tuning can also weaken previously applied safety behavior; red-teaming experiments showed that small fine-tuning datasets, including datasets without an explicitly malicious stated purpose, could produce substantial safety regressions in the tested aligned models.<ref name="Qi2023" />
Post-training can improve targeted behavior while degrading other properties. Fine-tuning can weaken previously applied safety behavior; red-teaming experiments have shown safety regressions after fine-tuning with small adversarial datasets and, to a lesser degree, with some benign datasets in the tested models.<ref name="Qi2023" /> Sequential stages can also cause capability regressions or catastrophic forgetting, depending on the data and objectives used.<ref name="Tie2025" /><ref name="Kumar2025" />


Preference-based methods depend on imperfect feedback. Human labels can be inconsistent, unrepresentative, or difficult to obtain for tasks whose correctness is hard to judge, while AI-generated labels depend on the labeler model, its prompts, and the criteria used to guide judgments.<ref name="Casper2023" /><ref name="Bai2022" /><ref name="Lee2023" /> A learned reward can be overoptimized: further improvement on the proxy reward may cease to correspond to improvement under a stronger or external evaluator.<ref name="Rafailov2024" /> This problem is related to reward hacking and motivates held-out evaluation, regularization, auditing, and the use of independently checkable outcomes when they are available.<ref name="Casper2023" /><ref name="Lambert2024" />
Preference-based methods depend on imperfect feedback. Human labels can be inconsistent, unrepresentative, or difficult to obtain for tasks whose correctness is hard to judge, while AI-generated labels depend on the labeler model and its instructions.<ref name="Kaufmann2025" /><ref name="Casper2023" /><ref name="Lee2024" /> Optimization can also overfit a learned or implicit preference signal: performance measured by the training objective may continue to improve while quality under an external evaluator plateaus or deteriorates.<ref name="Rafailov2024" />


Sequential training stages can interfere with one another. Surveys identify catastrophic forgetting, capability regressions, and trade-offs between alignment objectives and general benchmark performance as recurring concerns.<ref name="Kumar2025" /><ref name="Tie2025" /> Multi-stage pipelines also increase computational and engineering cost and make attribution difficult: a final behavior may depend on the base model, data curation, the order of stages, reward design, sampling, and evaluation choices.<ref name="Lambert2024" /><ref name="Zhao2026" /> Limited disclosure of post-training datasets and recipes can therefore hinder independent reproduction and comparison.<ref name="Lambert2024" />
Multi-stage pipelines increase computational and engineering cost and make causal attribution difficult. A final behavior may depend on the base model, data curation, stage order, reward design, sampling, and evaluation choices.<ref name="Lambert2024" /><ref name="Tie2025" /> Limited disclosure of post-training datasets and recipes can hinder independent reproduction and comparison.<ref name="Lambert2024" />


== See also ==
== See also ==