Archive:Training of large language models: Difference between revisions
Created sourced userspace draft on post-training methods for large language models |
Added arXiv classifications and disabled mainspace categories in userspace |
||
| Line 10: | Line 10: | ||
|title=A Survey on Post-training of Large Language Models | |title=A Survey on Post-training of Large Language Models | ||
|eprint=2503.06072 | |eprint=2503.06072 | ||
|class=cs.CL | |||
|year=2025 | |year=2025 | ||
}}</ref><ref name="Kumar2025">{{cite arXiv | }}</ref><ref name="Kumar2025">{{cite arXiv | ||
| Line 19: | Line 20: | ||
|title=LLM Post-Training: A Deep Dive into Reasoning Large Language Models | |title=LLM Post-Training: A Deep Dive into Reasoning Large Language Models | ||
|eprint=2502.21321 | |eprint=2502.21321 | ||
|class=cs.CL | |||
|year=2025 | |year=2025 | ||
}}</ref><ref name="Zhao2026">{{cite arXiv | }}</ref><ref name="Zhao2026">{{cite arXiv | ||
| Line 28: | Line 30: | ||
|title=Large Language Model Post-Training: A Unified View of Off-Policy and On-Policy Learning | |title=Large Language Model Post-Training: A Unified View of Off-Policy and On-Policy Learning | ||
|eprint=2604.07941 | |eprint=2604.07941 | ||
|class=cs.CL | |||
|year=2026 | |year=2026 | ||
}}</ref> Common post-training stages include [[Fine-tuning (deep learning)|supervised fine-tuning]], preference optimization, [[Reinforcement learning|reinforcement learning]], process or verifier-guided training, parameter-efficient adaptation, and [[Knowledge distillation|knowledge distillation]].<ref name="Tie2025" /><ref name="Zhao2026" /> | }}</ref> Common post-training stages include [[Fine-tuning (deep learning)|supervised fine-tuning]], preference optimization, [[Reinforcement learning|reinforcement learning]], process or verifier-guided training, parameter-efficient adaptation, and [[Knowledge distillation|knowledge distillation]].<ref name="Tie2025" /><ref name="Zhao2026" /> | ||
Post-training commonly uses datasets that are more structured than pretraining corpora and, in many published pipelines, substantially smaller. Depending on the method, the data may contain | Post-training commonly uses datasets that are more structured than pretraining corpora and, in many published pipelines, substantially smaller. Depending on the method, the data may contain instruction–response examples, comparisons between candidate responses, critiques generated by people or other models, intermediate reasoning steps, or outcomes that can be checked automatically.<ref name="Zhang2023">{{cite arXiv | ||
|last1=Zhang |first1=Shengyu | |last1=Zhang |first1=Shengyu | ||
|last2=Dong |first2=Linfeng | |last2=Dong |first2=Linfeng | ||
| Line 39: | Line 42: | ||
|title=Instruction Tuning for Large Language Models: A Survey | |title=Instruction Tuning for Large Language Models: A Survey | ||
|eprint=2308.10792 | |eprint=2308.10792 | ||
|class=cs.CL | |||
|year=2023 | |year=2023 | ||
}}</ref><ref name="Ouyang2022">{{cite arXiv | }}</ref><ref name="Ouyang2022">{{cite arXiv | ||
| Line 48: | Line 52: | ||
|title=Training language models to follow instructions with human feedback | |title=Training language models to follow instructions with human feedback | ||
|eprint=2203.02155 | |eprint=2203.02155 | ||
|class=cs.CL | |||
|year=2022 | |year=2022 | ||
}}</ref><ref name="Uesato2022">{{cite arXiv | }}</ref><ref name="Uesato2022">{{cite arXiv | ||
| Line 57: | Line 62: | ||
|title=Solving math word problems with process- and outcome-based feedback | |title=Solving math word problems with process- and outcome-based feedback | ||
|eprint=2211.14275 | |eprint=2211.14275 | ||
|class=cs.LG | |||
|year=2022 | |year=2022 | ||
}}</ref> Modern systems often use several stages in sequence rather than one objective throughout training.<ref name="Lambert2024">{{cite arXiv | }}</ref> Modern systems often use several stages in sequence rather than one objective throughout training.<ref name="Lambert2024">{{cite arXiv | ||
| Line 66: | Line 72: | ||
|title=Tulu 3: Pushing Frontiers in Open Language Model Post-Training | |title=Tulu 3: Pushing Frontiers in Open Language Model Post-Training | ||
|eprint=2411.15124 | |eprint=2411.15124 | ||
|class=cs.CL | |||
|year=2024 | |year=2024 | ||
}}</ref><ref name="DeepSeek2025">{{cite arXiv | }}</ref><ref name="DeepSeek2025">{{cite arXiv | ||
| Line 71: | Line 78: | ||
|title=DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning | |title=DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning | ||
|eprint=2501.12948 | |eprint=2501.12948 | ||
|class=cs.CL | |||
|year=2025 | |year=2025 | ||
}}</ref> | }}</ref> | ||
| Line 88: | Line 96: | ||
|series=Proceedings of Machine Learning Research | |series=Proceedings of Machine Learning Research | ||
|volume=119 | |volume=119 | ||
|pages= | |pages=7197–7206 | ||
|year=2020 | |year=2020 | ||
|url=https://proceedings.mlr.press/v119/nagel20a.html | |url=https://proceedings.mlr.press/v119/nagel20a.html | ||
| Line 105: | Line 113: | ||
|title=Finetuned Language Models Are Zero-Shot Learners | |title=Finetuned Language Models Are Zero-Shot Learners | ||
|eprint=2109.01652 | |eprint=2109.01652 | ||
|class=cs.CL | |||
|year=2021 | |year=2021 | ||
}}</ref> T0 similarly converted many supervised datasets into prompted tasks and trained a model to generalize to tasks that were excluded from its training mixture.<ref name="Sanh2021">{{cite arXiv | }}</ref> T0 similarly converted many supervised datasets into prompted tasks and trained a model to generalize to tasks that were excluded from its training mixture.<ref name="Sanh2021">{{cite arXiv | ||
| Line 114: | Line 123: | ||
|title=Multitask Prompted Training Enables Zero-Shot Task Generalization | |title=Multitask Prompted Training Enables Zero-Shot Task Generalization | ||
|eprint=2110.08207 | |eprint=2110.08207 | ||
|class=cs.LG | |||
|year=2021 | |year=2021 | ||
}}</ref> These projects helped establish instruction tuning as a way to bridge the next-token-prediction objective used in pretraining and the user-facing objective of responding to natural-language requests.<ref name="Zhang2023" /> | }}</ref> These projects helped establish instruction tuning as a way to bridge the next-token-prediction objective used in pretraining and the user-facing objective of responding to natural-language requests.<ref name="Zhang2023" /> | ||
| Line 125: | Line 135: | ||
|title=Learning to summarize from human feedback | |title=Learning to summarize from human feedback | ||
|eprint=2009.01325 | |eprint=2009.01325 | ||
|class=cs.CL | |||
|year=2020 | |year=2020 | ||
}}</ref> InstructGPT used a sequence of supervised fine-tuning on demonstrations, reward-model training on ranked outputs, and reinforcement learning from human feedback (RLHF).<ref name="Ouyang2022" /> Later RLHF systems and surveys retained this general sequence while identifying limitations in the collection, aggregation, and optimization of human feedback.<ref name="Casper2023">{{cite journal | }}</ref> InstructGPT used a sequence of supervised fine-tuning on demonstrations, reward-model training on ranked outputs, and reinforcement learning from human feedback (RLHF).<ref name="Ouyang2022" /> Later RLHF systems and surveys retained this general sequence while identifying limitations in the collection, aggregation, and optimization of human feedback.<ref name="Casper2023">{{cite journal | ||
| Line 146: | Line 157: | ||
|title=Direct Preference Optimization: Your Language Model is Secretly a Reward Model | |title=Direct Preference Optimization: Your Language Model is Secretly a Reward Model | ||
|eprint=2305.18290 | |eprint=2305.18290 | ||
|class=cs.LG | |||
|year=2023 | |year=2023 | ||
}}</ref> Other objectives broadened the forms of feedback that could be used. | }}</ref> Other objectives broadened the forms of feedback that could be used. Kahneman–Tversky optimization (KTO), for example, was designed to learn from a binary signal indicating whether an output was desirable rather than from paired rankings.<ref name="Ethayarajh2024">{{cite arXiv | ||
|last1=Ethayarajh |first1=Kawin | |last1=Ethayarajh |first1=Kawin | ||
|last2=Xu |first2=Winnie | |last2=Xu |first2=Winnie | ||
| Line 155: | Line 167: | ||
|title=KTO: Model Alignment as Prospect Theoretic Optimization | |title=KTO: Model Alignment as Prospect Theoretic Optimization | ||
|eprint=2402.01306 | |eprint=2402.01306 | ||
|class=cs.LG | |||
|year=2024 | |year=2024 | ||
}}</ref> | }}</ref> | ||
| Line 166: | Line 179: | ||
|title=Let's Verify Step by Step | |title=Let's Verify Step by Step | ||
|eprint=2305.20050 | |eprint=2305.20050 | ||
|class=cs.LG | |||
|year=2023 | |year=2023 | ||
}}</ref><ref name="DeepSeek2025" /> | }}</ref><ref name="DeepSeek2025" /> | ||
| Line 183: | Line 197: | ||
|title=Self-Instruct: Aligning Language Models with Self-Generated Instructions | |title=Self-Instruct: Aligning Language Models with Self-Generated Instructions | ||
|eprint=2212.10560 | |eprint=2212.10560 | ||
|class=cs.CL | |||
|year=2022 | |year=2022 | ||
}}</ref> Experiments with LIMA used 1,000 curated | }}</ref> Experiments with LIMA used 1,000 curated prompt–response pairs and argued that careful example selection can be important when a pretrained model already contains broad knowledge.<ref name="Zhou2023">{{cite arXiv | ||
|last1=Zhou |first1=Chunting | |last1=Zhou |first1=Chunting | ||
|last2=Liu |first2=Pengfei | |last2=Liu |first2=Pengfei | ||
| Line 192: | Line 207: | ||
|title=LIMA: Less Is More for Alignment | |title=LIMA: Less Is More for Alignment | ||
|eprint=2305.11206 | |eprint=2305.11206 | ||
|class=cs.CL | |||
|year=2023 | |year=2023 | ||
}}</ref> These results do not imply that a fixed amount of instruction data is sufficient for every model or task; instruction-tuning surveys report that outcomes depend on the model, task distribution, data construction, and training procedure.<ref name="Zhang2023" /> | }}</ref> These results do not imply that a fixed amount of instruction data is sufficient for every model or task; instruction-tuning surveys report that outcomes depend on the model, task distribution, data construction, and training procedure.<ref name="Zhang2023" /> | ||
| Line 205: | Line 221: | ||
A common RLHF pipeline begins with SFT, collects comparisons between model outputs, trains a reward model to predict those comparisons, and optimizes the language-model policy to increase the predicted reward.<ref name="Ouyang2022" /><ref name="Stiennon2020" /> The reward model is a learned proxy for the feedback data rather than a direct measurement of a universal human preference. InstructGPT, for example, described alignment relative to the preferences of its labelers and researchers, not to all possible users or value systems.<ref name="Ouyang2022" /> | A common RLHF pipeline begins with SFT, collects comparisons between model outputs, trains a reward model to predict those comparisons, and optimizes the language-model policy to increase the predicted reward.<ref name="Ouyang2022" /><ref name="Stiennon2020" /> The reward model is a learned proxy for the feedback data rather than a direct measurement of a universal human preference. InstructGPT, for example, described alignment relative to the preferences of its labelers and researchers, not to all possible users or value systems.<ref name="Ouyang2022" /> | ||
Reward models can be evaluated separately from the policies they train. RewardBench assembled | Reward models can be evaluated separately from the policies they train. RewardBench assembled prompt–response pairs covering chat, reasoning, and safety cases to compare reward models and DPO-style implicit reward functions.<ref name="RewardBench2024">{{cite arXiv | ||
|last1=Lambert |first1=Nathan | |last1=Lambert |first1=Nathan | ||
|last2=Pyatkin |first2=Valentina | |last2=Pyatkin |first2=Valentina | ||
| Line 213: | Line 229: | ||
|title=RewardBench: Evaluating Reward Models for Language Modeling | |title=RewardBench: Evaluating Reward Models for Language Modeling | ||
|eprint=2403.13787 | |eprint=2403.13787 | ||
|class=cs.LG | |||
|year=2024 | |year=2024 | ||
}}</ref> Such evaluation is relevant because a policy can exploit errors or blind spots in a learned reward model when optimization becomes stronger.<ref name="Casper2023" /><ref name="Rafailov2024">{{cite arXiv | }}</ref> Such evaluation is relevant because a policy can exploit errors or blind spots in a learned reward model when optimization becomes stronger.<ref name="Casper2023" /><ref name="Rafailov2024">{{cite arXiv | ||
| Line 222: | Line 239: | ||
|title=Scaling Laws for Reward Model Overoptimization in Direct Alignment Algorithms | |title=Scaling Laws for Reward Model Overoptimization in Direct Alignment Algorithms | ||
|eprint=2406.02900 | |eprint=2406.02900 | ||
|class=cs.LG | |||
|year=2024 | |year=2024 | ||
}}</ref> | }}</ref> | ||
| Line 241: | Line 259: | ||
|title=Constitutional AI: Harmlessness from AI Feedback | |title=Constitutional AI: Harmlessness from AI Feedback | ||
|eprint=2212.08073 | |eprint=2212.08073 | ||
|class=cs.CL | |||
|year=2022 | |year=2022 | ||
}}</ref> Reinforcement learning from AI feedback (RLAIF) trains a reward model using preferences generated by an AI labeler. In experiments on summarization and dialogue tasks, RLAIF produced results comparable to the studied RLHF baselines, while reducing dependence on human preference labels.<ref name="Lee2023">{{cite arXiv | }}</ref> Reinforcement learning from AI feedback (RLAIF) trains a reward model using preferences generated by an AI labeler. In experiments on summarization and dialogue tasks, RLAIF produced results comparable to the studied RLHF baselines, while reducing dependence on human preference labels.<ref name="Lee2023">{{cite arXiv | ||
| Line 250: | Line 269: | ||
|title=RLAIF vs. RLHF: Scaling Reinforcement Learning from Human Feedback with AI Feedback | |title=RLAIF vs. RLHF: Scaling Reinforcement Learning from Human Feedback with AI Feedback | ||
|eprint=2309.00267 | |eprint=2309.00267 | ||
|class=cs.CL | |||
|year=2023 | |year=2023 | ||
}}</ref> | }}</ref> | ||
| Line 266: | Line 286: | ||
|title=STaR: Bootstrapping Reasoning With Reasoning | |title=STaR: Bootstrapping Reasoning With Reasoning | ||
|eprint=2203.14465 | |eprint=2203.14465 | ||
|class=cs.LG | |||
|year=2022 | |year=2022 | ||
}}</ref> Research comparing process and outcome supervision found that feedback on intermediate reasoning steps can reduce reasoning errors that are not visible from the final answer alone.<ref name="Uesato2022" /> ''Let's Verify Step by Step'' trained process reward models on step-level judgments; in the studied mathematical-reasoning setting, those models outperformed outcome-supervised reward models.<ref name="Lightman2023" /> | }}</ref> Research comparing process and outcome supervision found that feedback on intermediate reasoning steps can reduce reasoning errors that are not visible from the final answer alone.<ref name="Uesato2022" /> ''Let's Verify Step by Step'' trained process reward models on step-level judgments; in the studied mathematical-reasoning setting, those models outperformed outcome-supervised reward models.<ref name="Lightman2023" /> | ||
| Line 277: | Line 298: | ||
|title=DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models | |title=DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models | ||
|eprint=2402.03300 | |eprint=2402.03300 | ||
|class=cs.CL | |||
|year=2024 | |year=2024 | ||
}}</ref> DeepSeek-R1 used a multi-stage pipeline that combined reinforcement learning, supervised fine-tuning on filtered reasoning data, further reinforcement learning, and distillation into smaller models.<ref name="DeepSeek2025" /> These systems illustrate one family of reasoning post-training; their reported results do not establish that the same recipe transfers unchanged to every domain or base model.<ref name="Kumar2025" /><ref name="Zhao2026" /> | }}</ref> DeepSeek-R1 used a multi-stage pipeline that combined reinforcement learning, supervised fine-tuning on filtered reasoning data, further reinforcement learning, and distillation into smaller models.<ref name="DeepSeek2025" /> These systems illustrate one family of reasoning post-training; their reported results do not establish that the same recipe transfers unchanged to every domain or base model.<ref name="Kumar2025" /><ref name="Zhao2026" /> | ||
| Line 290: | Line 312: | ||
|title=LoRA: Low-Rank Adaptation of Large Language Models | |title=LoRA: Low-Rank Adaptation of Large Language Models | ||
|eprint=2106.09685 | |eprint=2106.09685 | ||
|class=cs.CL | |||
|year=2021 | |year=2021 | ||
}}</ref> | }}</ref> | ||
| Line 300: | Line 323: | ||
|title=QLoRA: Efficient Finetuning of Quantized LLMs | |title=QLoRA: Efficient Finetuning of Quantized LLMs | ||
|eprint=2305.14314 | |eprint=2305.14314 | ||
|class=cs.LG | |||
|year=2023 | |year=2023 | ||
}}</ref> LoRA and QLoRA describe how an update is parameterized; they can be used with instruction-tuning or other post-training objectives rather than constituting a separate behavioral goal.<ref name="Tie2025" /> | }}</ref> LoRA and QLoRA describe how an update is parameterized; they can be used with instruction-tuning or other post-training objectives rather than constituting a separate behavioral goal.<ref name="Tie2025" /> | ||
| Line 305: | Line 329: | ||
=== Distillation and consolidation === | === Distillation and consolidation === | ||
[[Knowledge distillation]] trains a smaller or otherwise different student model to reproduce behavior from a teacher model. In LLM post-training, the transferred signal can be token probabilities, generated responses, ranked outputs, reasoning traces, or preferences.<ref name="Zhao2026" /> MiniLLM used an on-policy distillation objective based on reverse | [[Knowledge distillation]] trains a smaller or otherwise different student model to reproduce behavior from a teacher model. In LLM post-training, the transferred signal can be token probabilities, generated responses, ranked outputs, reasoning traces, or preferences.<ref name="Zhao2026" /> MiniLLM used an on-policy distillation objective based on reverse Kullback–Leibler divergence to train smaller generative language models from larger teachers.<ref name="Gu2023">{{cite arXiv | ||
|last1=Gu |first1=Yuxian | |last1=Gu |first1=Yuxian | ||
|last2=Dong |first2=Li | |last2=Dong |first2=Li | ||
| Line 312: | Line 336: | ||
|title=MiniLLM: On-Policy Distillation of Large Language Models | |title=MiniLLM: On-Policy Distillation of Large Language Models | ||
|eprint=2306.08543 | |eprint=2306.08543 | ||
|class=cs.CL | |||
|year=2023 | |year=2023 | ||
}}</ref> Zephyr combined distilled SFT data with preference data ranked by a teacher model and applied distilled DPO.<ref name="Tunstall2023">{{cite arXiv | }}</ref> Zephyr combined distilled SFT data with preference data ranked by a teacher model and applied distilled DPO.<ref name="Tunstall2023">{{cite arXiv | ||
| Line 321: | Line 346: | ||
|title=Zephyr: Direct Distillation of LM Alignment | |title=Zephyr: Direct Distillation of LM Alignment | ||
|eprint=2310.16944 | |eprint=2310.16944 | ||
|class=cs.LG | |||
|year=2023 | |year=2023 | ||
}}</ref> | }}</ref> | ||
| Line 338: | Line 364: | ||
|title=Fine-tuning Aligned Language Models Compromises Safety, Even When Users Do Not Intend To! | |title=Fine-tuning Aligned Language Models Compromises Safety, Even When Users Do Not Intend To! | ||
|eprint=2310.03693 | |eprint=2310.03693 | ||
|class=cs.CL | |||
|year=2023 | |year=2023 | ||
}}</ref> Multi-stage recipes may use development benchmarks during training and separate unseen evaluations afterward. Tulu 3 also applied benchmark decontamination to reduce overlap between training examples and evaluation data.<ref name="Lambert2024" /> | }}</ref> Multi-stage recipes may use development benchmarks during training and separate unseen evaluations afterward. Tulu 3 also applied benchmark decontamination to reduce overlap between training examples and evaluation data.<ref name="Lambert2024" /> | ||
| Line 349: | Line 376: | ||
|title=Understanding the Effects of RLHF on LLM Generalisation and Diversity | |title=Understanding the Effects of RLHF on LLM Generalisation and Diversity | ||
|eprint=2310.06452 | |eprint=2310.06452 | ||
|class=cs.LG | |||
|year=2023 | |year=2023 | ||
}}</ref><ref name="Casper2023" /> Comparisons are also sensitive to the base model, training data, prompts, decoding settings, and evaluator, so results from one pipeline do not by themselves identify a universally preferable method.<ref name="Zhang2023" /><ref name="Ethayarajh2024" /><ref name="Zhao2026" /> | }}</ref><ref name="Casper2023" /> Comparisons are also sensitive to the base model, training data, prompts, decoding settings, and evaluator, so results from one pipeline do not by themselves identify a universally preferable method.<ref name="Zhang2023" /><ref name="Ethayarajh2024" /><ref name="Zhao2026" /> | ||
| Line 374: | Line 402: | ||
{{Reflist}} | {{Reflist}} | ||
<!-- Categories to add after the move: | |||
[[Category:Large language models]] | [[Category:Large language models]] | ||
[[Category:Natural language processing]] | [[Category:Natural language processing]] | ||
--> | |||
Revision as of 16:35, 12 July 2026
Template:Short description Template:Use American English
Post-training of large language models is the set of training methods applied to a large language model (LLM) after its initial large-scale pretraining. Post-training starts from a pretrained or base model and changes its behavior for purposes such as following instructions, matching stated preferences, improving multi-step reasoning, adapting to a domain, or transferring capabilities into a smaller or more efficient model.[1][2][3] Common post-training stages include supervised fine-tuning, preference optimization, reinforcement learning, process or verifier-guided training, parameter-efficient adaptation, and knowledge distillation.[1][3]
Post-training commonly uses datasets that are more structured than pretraining corpora and, in many published pipelines, substantially smaller. Depending on the method, the data may contain instruction–response examples, comparisons between candidate responses, critiques generated by people or other models, intermediate reasoning steps, or outcomes that can be checked automatically.[4][5][6] Modern systems often use several stages in sequence rather than one objective throughout training.[7][8]
Scope and terminology
The boundary of post-training is not identical across the literature. Dedicated surveys consistently include supervised fine-tuning, preference-based alignment, reinforcement learning, and reasoning-oriented training, while some broader taxonomies also include efficiency methods, multimodal adaptation, tool integration, and inference-time scaling.[1][2][3] Some surveys distinguish weight-updating post-training from inference-time scaling, while broader surveys discuss both within a single account of how pretrained models are adapted.[2][3]
The term is also distinct from post-training quantization, which reduces the numerical precision of an already trained network without referring to the broader process of teaching instruction following, preferences, or reasoning behavior.[9][1]
Post-training does not necessarily correspond to a single objective. A 2026 survey organizes methods by whether they learn from externally supplied trajectories or from the model's own rollouts, and describes their roles as expanding reachable behavior, reshaping probabilities among reachable behaviors, or consolidating behavior across stages and model sizes.[3] This view treats multi-stage recipes as coordinated systems rather than as interchangeable fine-tuning procedures.
Development
Multitask instruction tuning was among the early methods later grouped under LLM post-training. FLAN fine-tuned a pretrained model on instructions for more than 60 natural-language-processing tasks and improved zero-shot performance on held-out task types.[10] T0 similarly converted many supervised datasets into prompted tasks and trained a model to generalize to tasks that were excluded from its training mixture.[11] These projects helped establish instruction tuning as a way to bridge the next-token-prediction objective used in pretraining and the user-facing objective of responding to natural-language requests.[4]
Preference-based post-training developed alongside instruction tuning. Work on abstractive summarization trained a reward model from human comparisons and then optimized a language model against that reward.[12] InstructGPT used a sequence of supervised fine-tuning on demonstrations, reward-model training on ranked outputs, and reinforcement learning from human feedback (RLHF).[5] Later RLHF systems and surveys retained this general sequence while identifying limitations in the collection, aggregation, and optimization of human feedback.[13]
Direct preference optimization (DPO), introduced in 2023, reformulated a commonly used preference-learning objective as a classification loss and removed the need to train a separate reward model and run an online reinforcement-learning loop.[14] Other objectives broadened the forms of feedback that could be used. Kahneman–Tversky optimization (KTO), for example, was designed to learn from a binary signal indicating whether an output was desirable rather than from paired rankings.[15]
Post-training recipes increasingly combined several data sources and objectives. The open Tulu 3 recipe used supervised fine-tuning, DPO, and reinforcement learning with automatically verifiable rewards, together with dataset curation, decontamination, and separate development and unseen evaluations.[7] Reasoning-focused pipelines also combined generated rationales, process supervision, rule-based or learned verifiers, reinforcement learning, and distillation.[6][16][8]
Methods
Supervised fine-tuning and instruction tuning
In supervised fine-tuning (SFT), the model is trained to predict a target response for a given input. Instruction tuning is a form of SFT in which examples are organized as natural-language instructions paired with desired outputs. Surveys of instruction tuning examine the construction of instruction datasets, the generation or collection of outputs, training mixtures, and the effects of data quantity and quality.[4]
Instruction data can be written by people, converted from existing task datasets, generated synthetically, or assembled from several sources. Self-Instruct used a language model to generate instructions, inputs, and outputs, filtered invalid or near-duplicate examples, and then used the resulting data to fine-tune the model.[17] Experiments with LIMA used 1,000 curated prompt–response pairs and argued that careful example selection can be important when a pretrained model already contains broad knowledge.[18] These results do not imply that a fixed amount of instruction data is sufficient for every model or task; instruction-tuning surveys report that outcomes depend on the model, task distribution, data construction, and training procedure.[4]
SFT is often the first stage of a longer post-training pipeline. It can establish response formats and basic instruction following before preference optimization or reinforcement learning, and it can provide a starting policy that generates outputs for later comparison or verification.[5][7][3]
Preference learning and alignment
Preference learning uses judgments about candidate outputs rather than only a single target answer. Feedback can be collected as rankings, pairwise comparisons, or labels such as desirable and undesirable. The training objective then increases the likelihood of preferred behavior while usually constraining the updated model from moving too far from a reference policy.[5][14][15]
Reward modeling and RLHF
A common RLHF pipeline begins with SFT, collects comparisons between model outputs, trains a reward model to predict those comparisons, and optimizes the language-model policy to increase the predicted reward.[5][12] The reward model is a learned proxy for the feedback data rather than a direct measurement of a universal human preference. InstructGPT, for example, described alignment relative to the preferences of its labelers and researchers, not to all possible users or value systems.[5]
Reward models can be evaluated separately from the policies they train. RewardBench assembled prompt–response pairs covering chat, reasoning, and safety cases to compare reward models and DPO-style implicit reward functions.[19] Such evaluation is relevant because a policy can exploit errors or blind spots in a learned reward model when optimization becomes stronger.[13][20]
Direct preference optimization
DPO learns directly from preferred and rejected responses. Its derivation expresses the optimal policy for a class of reward-regularized objectives in a form that can be fitted with a binary classification loss, avoiding a separately trained reward model and an online policy-gradient stage.[14] DPO nevertheless depends on the coverage and quality of the comparison data and on choices such as the reference policy and regularization strength.[14][20]
Related direct-alignment objectives use different feedback assumptions. KTO was proposed for settings in which examples are labeled desirable or undesirable without a matched alternative response.[15] These objectives are alternatives within preference-based post-training rather than evidence that one loss is uniformly best across datasets and applications.[15][3]
AI-generated feedback
Feedback can also be produced or assisted by another language model. Constitutional AI used a written set of principles to guide model-generated critiques and revisions, followed by preference learning from AI feedback.[21] Reinforcement learning from AI feedback (RLAIF) trains a reward model using preferences generated by an AI labeler. In experiments on summarization and dialogue tasks, RLAIF produced results comparable to the studied RLHF baselines, while reducing dependence on human preference labels.[22]
AI-generated feedback reduces dependence on human preference labels, but its results depend on the capabilities and prompts of the labeler model, the written principles used to guide judgments, and separate validation procedures.[21][22][13]
Reasoning-focused post-training
Reasoning-oriented post-training uses examples or rewards that distinguish successful multi-step problem solving from plausible but incorrect output. Training signals can include worked solutions, model-generated rationales that pass a verifier, feedback on intermediate steps, or rewards based on a checkable final answer.[2][6][16]
The Self-Taught Reasoner (STaR) iteratively generated rationales, retained rationales that led to correct answers, and fine-tuned on the retained examples.[23] Research comparing process and outcome supervision found that feedback on intermediate reasoning steps can reduce reasoning errors that are not visible from the final answer alone.[6] Let's Verify Step by Step trained process reward models on step-level judgments; in the studied mathematical-reasoning setting, those models outperformed outcome-supervised reward models.[16]
When outcomes can be checked automatically, reinforcement learning can use rule-based or programmatic rewards instead of subjective preference labels. DeepSeekMath introduced group relative policy optimization (GRPO), a variant of proximal policy optimization designed to improve mathematical reasoning while reducing the memory requirements of policy optimization.[24] DeepSeek-R1 used a multi-stage pipeline that combined reinforcement learning, supervised fine-tuning on filtered reasoning data, further reinforcement learning, and distillation into smaller models.[8] These systems illustrate one family of reasoning post-training; their reported results do not establish that the same recipe transfers unchanged to every domain or base model.[2][3]
Parameter-efficient adaptation
Full fine-tuning updates all model parameters and can require substantial accelerator memory. Parameter-efficient methods instead train a smaller set of added or selected parameters while leaving most base-model weights fixed. Low-rank adaptation (LoRA) represents weight updates with trainable low-rank matrices, reducing the number of trainable parameters and the memory needed to store separate task adaptations.[25]
Quantized low-rank adaptation (QLoRA) combines low-rank adapters with a quantized frozen base model. Its design backpropagates gradients through a frozen 4-bit quantized model into LoRA adapters, reducing memory use during fine-tuning.[26] LoRA and QLoRA describe how an update is parameterized; they can be used with instruction-tuning or other post-training objectives rather than constituting a separate behavioral goal.[1]
Distillation and consolidation
Knowledge distillation trains a smaller or otherwise different student model to reproduce behavior from a teacher model. In LLM post-training, the transferred signal can be token probabilities, generated responses, ranked outputs, reasoning traces, or preferences.[3] MiniLLM used an on-policy distillation objective based on reverse Kullback–Leibler divergence to train smaller generative language models from larger teachers.[27] Zephyr combined distilled SFT data with preference data ranked by a teacher model and applied distilled DPO.[28]
Distillation can serve both deployment efficiency and behavioral consolidation. A teacher's post-trained behavior may be transferred into a smaller model, or outputs from a reasoning-oriented model may become SFT data for another model.[3][8] The resulting student behavior depends on the teacher outputs and on the coverage and filtering of the distillation dataset.[27][28]
Data and evaluation
Post-training datasets differ by stage. SFT uses target responses; preference optimization uses comparisons or desirability labels; process supervision uses judgments on intermediate steps; and reinforcement learning with verifiable rewards uses outcomes that can be checked by rules, tests, or formal answer matching.[4][5][6][7] Synthetic data can expand coverage, but filtering is needed to remove invalid, repetitive, or low-quality generations.[17][23]
Evaluation is normally matched to the intended effect of each stage. Instruction following can be assessed on held-out tasks and human judgments; reward models can be compared on preference test sets; reasoning systems can be evaluated using answer correctness and, when available, step-level judgments; and safety-oriented post-training requires separate adversarial or policy-relevant tests.[10][19][16][29] Multi-stage recipes may use development benchmarks during training and separate unseen evaluations afterward. Tulu 3 also applied benchmark decontamination to reduce overlap between training examples and evaluation data.[7]
No single score measures all post-training goals. Improvements in instruction following, reward-model accuracy, reasoning correctness, diversity, harmlessness, or domain performance can move differently under the same update.[30][13] Comparisons are also sensitive to the base model, training data, prompts, decoding settings, and evaluator, so results from one pipeline do not by themselves identify a universally preferable method.[4][15][3]
Limitations
Post-training can improve targeted behavior while degrading other properties. Experiments comparing SFT and RLHF found a trade-off in the studied settings: RLHF improved out-of-distribution generalization relative to SFT but reduced output diversity.[30] Fine-tuning can also weaken previously applied safety behavior; red-teaming experiments showed that small fine-tuning datasets, including datasets without an explicitly malicious stated purpose, could produce substantial safety regressions in the tested aligned models.[29]
Preference-based methods depend on imperfect feedback. Human labels can be inconsistent, unrepresentative, or difficult to obtain for tasks whose correctness is hard to judge, while AI-generated labels depend on the labeler model, its prompts, and the criteria used to guide judgments.[13][21][22] A learned reward can be overoptimized: further improvement on the proxy reward may cease to correspond to improvement under a stronger or external evaluator.[20] This problem is related to reward hacking and motivates held-out evaluation, regularization, auditing, and the use of independently checkable outcomes when they are available.[13][7]
Sequential training stages can interfere with one another. Surveys identify catastrophic forgetting, capability regressions, and trade-offs between alignment objectives and general benchmark performance as recurring concerns.[2][1] Multi-stage pipelines also increase computational and engineering cost and make attribution difficult: a final behavior may depend on the base model, data curation, the order of stages, reward design, sampling, and evaluation choices.[7][3] Limited disclosure of post-training datasets and recipes can therefore hinder independent reproduction and comparison.[7]
See also
- AI alignment
- Fine-tuning (deep learning)
- Knowledge distillation
- Large language model
- LoRA (machine learning)
- Reasoning model
- Reinforcement learning from human feedback
References
- ↑ 1.0 1.1 1.2 1.3 1.4 1.5 Template:Cite arXiv
- ↑ 2.0 2.1 2.2 2.3 2.4 2.5 Template:Cite arXiv
- ↑ 3.00 3.01 3.02 3.03 3.04 3.05 3.06 3.07 3.08 3.09 3.10 3.11 Template:Cite arXiv
- ↑ 4.0 4.1 4.2 4.3 4.4 4.5 Template:Cite arXiv
- ↑ 5.0 5.1 5.2 5.3 5.4 5.5 5.6 Template:Cite arXiv
- ↑ 6.0 6.1 6.2 6.3 6.4 Template:Cite arXiv
- ↑ 7.0 7.1 7.2 7.3 7.4 7.5 7.6 7.7 Template:Cite arXiv
- ↑ 8.0 8.1 8.2 8.3 Template:Cite arXiv
- ↑ Template:Cite conference
- ↑ 10.0 10.1 Template:Cite arXiv
- ↑ Template:Cite arXiv
- ↑ 12.0 12.1 Template:Cite arXiv
- ↑ 13.0 13.1 13.2 13.3 13.4 13.5 Casper, Stephen; Davies, Xander; Shi, Claudia; Gilbert, Thomas Krendl (18 September 2023). "Open Problems and Fundamental Limitations of Reinforcement Learning from Human Feedback". Transactions on Machine Learning Research.
- ↑ 14.0 14.1 14.2 14.3 Template:Cite arXiv
- ↑ 15.0 15.1 15.2 15.3 15.4 Template:Cite arXiv
- ↑ 16.0 16.1 16.2 16.3 Template:Cite arXiv
- ↑ 17.0 17.1 Template:Cite arXiv
- ↑ Template:Cite arXiv
- ↑ 19.0 19.1 Template:Cite arXiv
- ↑ 20.0 20.1 20.2 Template:Cite arXiv
- ↑ 21.0 21.1 21.2 Template:Cite arXiv
- ↑ 22.0 22.1 22.2 Template:Cite arXiv
- ↑ 23.0 23.1 Template:Cite arXiv
- ↑ Template:Cite arXiv
- ↑ Template:Cite arXiv
- ↑ Template:Cite arXiv
- ↑ 27.0 27.1 Template:Cite arXiv
- ↑ 28.0 28.1 Template:Cite arXiv
- ↑ 29.0 29.1 Template:Cite arXiv
- ↑ 30.0 30.1 Template:Cite arXiv