Archive:Training of large language models: Difference between revisions
Added arXiv classifications and disabled mainspace categories in userspace |
Reworked references to use final journal and conference publications where available |
||
| Line 2: | Line 2: | ||
{{Use American English|date=July 2026}} | {{Use American English|date=July 2026}} | ||
'''Post-training of large language models''' is | '''Post-training of large language models''' is an umbrella term for training applied to a [[Large language model|large language model]] (LLM) after its initial large-scale pretraining. It starts from a pretrained or ''base'' model and changes its behavior for purposes such as following instructions, matching stated preferences, adapting to a domain, improving multi-step reasoning, or transferring behavior to a smaller model.<ref name="Tie2025">{{cite arXiv | ||
|last1=Tie |first1=Guiyao | |last1=Tie |first1=Guiyao | ||
|last2=Zhao |first2=Zeli | |last2=Zhao |first2=Zeli | ||
| Line 22: | Line 22: | ||
|class=cs.CL | |class=cs.CL | ||
|year=2025 | |year=2025 | ||
}}</ref> Common components include [[Fine-tuning (deep learning)|supervised fine-tuning]], instruction tuning, preference optimization, [[Reinforcement learning|reinforcement learning]], process or verifier-guided training, parameter-efficient adaptation, and [[Knowledge distillation|knowledge distillation]].<ref name="Zhang2023">{{cite arXiv | |||
}}</ref> Common | |||
|last1=Zhang |first1=Shengyu | |last1=Zhang |first1=Shengyu | ||
|last2=Dong |first2=Linfeng | |last2=Dong |first2=Linfeng | ||
| Line 44: | Line 32: | ||
|class=cs.CL | |class=cs.CL | ||
|year=2023 | |year=2023 | ||
}}</ref><ref name="Ouyang2022">{{cite | }}</ref><ref name="Kaufmann2025">{{cite journal | ||
|last1=Kaufmann |first1=Timo | |||
|last2=Weng |first2=Paul | |||
|last3=Bengs |first3=Viktor | |||
|last4=Hüllermeier |first4=Eyke | |||
|title=A Survey of Reinforcement Learning from Human Feedback | |||
|journal=Transactions on Machine Learning Research | |||
|year=2025 | |||
|arxiv=2312.14925 | |||
}}</ref> | |||
Post-training data are usually more structured than pretraining corpora. Depending on the method, examples may contain instructions paired with target responses, comparisons between candidate responses, labels indicating desirable or undesirable output, model-generated critiques, intermediate reasoning steps, or outcomes that can be checked automatically.<ref name="Chung2024">{{cite journal | |||
|last1=Chung |first1=Hyung Won | |||
|last2=Hou |first2=Le | |||
|last3=Longpre |first3=Shayne | |||
|last4=Zoph |first4=Barret | |||
|display-authors=3 | |||
|title=Scaling Instruction-Finetuned Language Models | |||
|journal=Journal of Machine Learning Research | |||
|volume=25 | |||
|issue=70 | |||
|pages=1–53 | |||
|year=2024 | |||
|arxiv=2210.11416 | |||
}}</ref><ref name="Ouyang2022">{{cite conference | |||
|last1=Ouyang |first1=Long | |last1=Ouyang |first1=Long | ||
|last2=Wu |first2=Jeff | |last2=Wu |first2=Jeff | ||
| Line 50: | Line 62: | ||
|last4=Almeida |first4=Diogo | |last4=Almeida |first4=Diogo | ||
|display-authors=3 | |display-authors=3 | ||
|title=Training | |title=Training Language Models to Follow Instructions with Human Feedback | ||
| | |book-title=Advances in Neural Information Processing Systems 35 | ||
|year=2022 | |year=2022 | ||
|arxiv=2203.02155 | |||
}}</ref> Practical systems often combine several stages rather than optimize one objective throughout training.<ref name="Lambert2024">{{cite arXiv | |||
| | |||
}}</ref> | |||
|last1=Lambert |first1=Nathan | |last1=Lambert |first1=Nathan | ||
|last2=Morrison |first2=Jacob | |last2=Morrison |first2=Jacob | ||
| Line 74: | Line 76: | ||
|class=cs.CL | |class=cs.CL | ||
|year=2024 | |year=2024 | ||
}}</ref> | }}</ref> | ||
== Scope and terminology == | == Scope and terminology == | ||
The boundary of ''post-training'' | The boundary of ''post-training'' varies across the literature. Recent surveys use it for overlapping sets of methods centered on adapting model behavior after pretraining, but they differ on whether to include topics such as multimodal adaptation, tool use, efficiency methods, and inference-time scaling.<ref name="Tie2025" /><ref name="Kumar2025" /> More established literatures treat instruction tuning and reinforcement learning from human feedback (RLHF) as distinct but frequently combined components of model adaptation.<ref name="Zhang2023" /><ref name="Kaufmann2025" /><ref name="Chung2024" /> | ||
The term is | The term is distinct from [[Large language model#Quantization|post-training quantization]], which reduces the numerical precision of an already trained network without referring to the broader process of teaching instruction following, preferences, or reasoning behavior.<ref name="Nagel2020">{{cite conference | ||
|last1=Nagel |first1=Markus | |last1=Nagel |first1=Markus | ||
|last2=Amjad |first2=Rana Ali | |last2=Amjad |first2=Rana Ali | ||
| Line 99: | Line 95: | ||
|year=2020 | |year=2020 | ||
|url=https://proceedings.mlr.press/v119/nagel20a.html | |url=https://proceedings.mlr.press/v119/nagel20a.html | ||
}}</ref | }}</ref> | ||
Post-training does not | Post-training also does not imply a single objective. A pipeline may first establish basic instruction-following behavior with supervised examples, then train on preference data, and finally use reinforcement learning or distillation for a narrower capability or deployment target.<ref name="Ouyang2022" /><ref name="Lambert2024" /> | ||
== Development == | == Development == | ||
Multitask instruction tuning was | Multitask instruction tuning was an early method later grouped within LLM post-training. T0 converted many supervised datasets into prompted tasks and trained a model to generalize to task types excluded from its training mixture.<ref name="Sanh2022">{{cite conference | ||
|last1=Sanh |first1=Victor | |last1=Sanh |first1=Victor | ||
|last2=Webson |first2=Albert | |last2=Webson |first2=Albert | ||
| Line 122: | Line 108: | ||
|display-authors=3 | |display-authors=3 | ||
|title=Multitask Prompted Training Enables Zero-Shot Task Generalization | |title=Multitask Prompted Training Enables Zero-Shot Task Generalization | ||
| | |book-title=International Conference on Learning Representations | ||
| | |year=2022 | ||
| | |arxiv=2110.08207 | ||
}}</ref> | }}</ref> Later FLAN work scaled instruction tuning across model families, task collections, model sizes, and chain-of-thought data.<ref name="Chung2024" /> | ||
Preference-based post-training developed alongside instruction tuning. Work on abstractive summarization trained a reward model from human comparisons and then optimized a language model against that reward.<ref name="Stiennon2020">{{cite | Preference-based post-training developed alongside instruction tuning. Work on abstractive summarization trained a reward model from human comparisons and then optimized a language model against that reward.<ref name="Stiennon2020">{{cite conference | ||
|last1=Stiennon |first1=Nisan | |last1=Stiennon |first1=Nisan | ||
|last2=Ouyang |first2=Long | |last2=Ouyang |first2=Long | ||
| Line 133: | Line 119: | ||
|last4=Ziegler |first4=Daniel M. | |last4=Ziegler |first4=Daniel M. | ||
|display-authors=3 | |display-authors=3 | ||
|title=Learning to | |title=Learning to Summarize from Human Feedback | ||
| | |book-title=Advances in Neural Information Processing Systems 33 | ||
|year=2020 | |year=2020 | ||
}}</ref> InstructGPT used | |arxiv=2009.01325 | ||
}}</ref> InstructGPT used supervised fine-tuning on demonstrations, reward-model training on ranked outputs, and reinforcement learning from human feedback.<ref name="Ouyang2022" /> Reviews of RLHF describe the approach as combining human-feedback collection, reward learning, and policy optimization, while also identifying limitations in each stage.<ref name="Kaufmann2025" /><ref name="Casper2023">{{cite journal | |||
|last1=Casper |first1=Stephen | |last1=Casper |first1=Stephen | ||
|last2=Davies |first2=Xander | |last2=Davies |first2=Xander | ||
| Line 149: | Line 135: | ||
}}</ref> | }}</ref> | ||
Direct preference optimization (DPO), introduced in 2023, reformulated a commonly used preference-learning objective as a classification loss | Direct preference optimization (DPO), introduced in 2023, reformulated a commonly used preference-learning objective as a classification loss, avoiding a separately trained reward model and an online policy-gradient stage.<ref name="Rafailov2023">{{cite conference | ||
|last1=Rafailov |first1=Rafael | |last1=Rafailov |first1=Rafael | ||
|last2=Sharma |first2=Archit | |last2=Sharma |first2=Archit | ||
| Line 155: | Line 141: | ||
|last4=Ermon |first4=Stefano | |last4=Ermon |first4=Stefano | ||
|display-authors=3 | |display-authors=3 | ||
|title=Direct Preference Optimization: Your Language Model | |title=Direct Preference Optimization: Your Language Model Is Secretly a Reward Model | ||
| | |book-title=Advances in Neural Information Processing Systems 36 | ||
|year=2023 | |year=2023 | ||
}}</ref> | |arxiv=2305.18290 | ||
}}</ref> Kahneman-Tversky optimization (KTO) was designed for data labeled desirable or undesirable without requiring a matched pair of responses.<ref name="Ethayarajh2024">{{cite conference | |||
|last1=Ethayarajh |first1=Kawin | |last1=Ethayarajh |first1=Kawin | ||
|last2=Xu |first2=Winnie | |last2=Xu |first2=Winnie | ||
| Line 166: | Line 152: | ||
|display-authors=3 | |display-authors=3 | ||
|title=KTO: Model Alignment as Prospect Theoretic Optimization | |title=KTO: Model Alignment as Prospect Theoretic Optimization | ||
| | |book-title=Proceedings of the 41st International Conference on Machine Learning | ||
| | |series=Proceedings of Machine Learning Research | ||
|volume=235 | |||
|year=2024 | |year=2024 | ||
|arxiv=2402.01306 | |||
}}</ref> | }}</ref> | ||
== Methods == | == Methods == | ||
| Line 187: | Line 163: | ||
=== Supervised fine-tuning and instruction tuning === | === Supervised fine-tuning and instruction tuning === | ||
In supervised fine-tuning (SFT), | In supervised fine-tuning (SFT), a model is trained to predict a target response for a given input. Instruction tuning is a form of SFT in which examples are organized as natural-language instructions paired with desired outputs. Research on instruction tuning examines task mixtures, instruction and response construction, model scale, data quality, and generalization to unseen tasks.<ref name="Zhang2023" /><ref name="Chung2024" /> | ||
Instruction data can be written by people, converted from existing task datasets, generated synthetically, or assembled from several sources. Self-Instruct used a language model to generate instructions, inputs, and outputs, filtered invalid or near-duplicate examples, and then used the resulting data | Instruction data can be written by people, converted from existing task datasets, generated synthetically, or assembled from several sources. Self-Instruct used a language model to generate instructions, inputs, and outputs, filtered invalid or near-duplicate examples, and then used the resulting data for fine-tuning.<ref name="Wang2023">{{cite conference | ||
|last1=Wang |first1=Yizhong | |last1=Wang |first1=Yizhong | ||
|last2=Kordi |first2=Yeganeh | |last2=Kordi |first2=Yeganeh | ||
| Line 196: | Line 172: | ||
|display-authors=3 | |display-authors=3 | ||
|title=Self-Instruct: Aligning Language Models with Self-Generated Instructions | |title=Self-Instruct: Aligning Language Models with Self-Generated Instructions | ||
| | |book-title=Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics | ||
|year=2023 | |year=2023 | ||
}}</ref> | |arxiv=2212.10560 | ||
}}</ref> Results from particular instruction-tuning recipes do not imply that a fixed amount or composition of data is sufficient for every model or task; outcomes depend on the base model, task distribution, data construction, and training procedure.<ref name="Zhang2023" /><ref name="Chung2024" /> | |||
SFT is often the first stage of a longer | SFT is often the first stage of a longer pipeline. It can establish response formats and basic instruction following before preference optimization or reinforcement learning, and it can provide a starting policy that generates outputs for later comparison or verification.<ref name="Ouyang2022" /><ref name="Lambert2024" /> | ||
=== Preference learning and alignment === | === Preference learning and alignment === | ||
Preference learning uses judgments about candidate outputs rather than only a single target answer. Feedback can be collected as rankings, pairwise comparisons, or labels such as desirable and undesirable. The training objective then increases the | Preference learning uses judgments about candidate outputs rather than only a single target answer. Feedback can be collected as rankings, pairwise comparisons, or labels such as desirable and undesirable. The training objective then increases the probability of preferred behavior while commonly constraining the updated model from moving too far from a reference policy.<ref name="Kaufmann2025" /><ref name="Ouyang2022" /><ref name="Rafailov2023" /> | ||
==== Reward modeling and RLHF ==== | ==== Reward modeling and RLHF ==== | ||
A common RLHF pipeline begins with SFT, collects comparisons between model outputs, trains a reward model to predict those comparisons, and optimizes the language-model policy to increase the predicted reward.<ref name="Ouyang2022" /><ref name="Stiennon2020" /> The reward model is a learned proxy for the feedback | A common RLHF pipeline begins with SFT, collects comparisons between model outputs, trains a reward model to predict those comparisons, and optimizes the language-model policy to increase the predicted reward.<ref name="Ouyang2022" /><ref name="Stiennon2020" /><ref name="Kaufmann2025" /> The reward model is a learned proxy for the collected feedback rather than a direct measurement of a universal human preference. The behavior learned by the policy therefore depends on the annotators, instructions, sampling process, and aggregation method used to create the feedback data.<ref name="Ouyang2022" /><ref name="Casper2023" /> | ||
==== Direct preference optimization ==== | |||
DPO learns directly from preferred and rejected responses. Its derivation expresses the optimal policy for a class of reward-regularized objectives in a form that can be fitted with a binary classification loss, eliminating the separately trained reward model and online reinforcement-learning loop used in a common RLHF pipeline.<ref name="Rafailov2023" /> DPO nevertheless depends on the coverage and quality of comparison data and on choices such as the reference policy and regularization strength.<ref name="Rafailov2023" /><ref name="Rafailov2024">{{cite conference | |||
|last1=Rafailov |first1=Rafael | |last1=Rafailov |first1=Rafael | ||
|last2=Chittepu |first2=Yaswanth | |last2=Chittepu |first2=Yaswanth | ||
| Line 238: | Line 196: | ||
|display-authors=3 | |display-authors=3 | ||
|title=Scaling Laws for Reward Model Overoptimization in Direct Alignment Algorithms | |title=Scaling Laws for Reward Model Overoptimization in Direct Alignment Algorithms | ||
| | |book-title=Advances in Neural Information Processing Systems 37 | ||
|year=2024 | |year=2024 | ||
|arxiv=2406.02900 | |||
}}</ref> | }}</ref> | ||
Related direct-alignment objectives use different assumptions about feedback. KTO, for example, was proposed for examples labeled desirable or undesirable without a matched alternative response.<ref name="Ethayarajh2024" /> These objectives are alternatives within preference-based post-training rather than evidence that one loss is uniformly best across datasets and applications.<ref name="Ethayarajh2024" /><ref name="Casper2023" /> | |||
Related direct-alignment objectives use different feedback | |||
==== AI-generated feedback ==== | ==== AI-generated feedback ==== | ||
Feedback can | Feedback can be produced or assisted by another language model. Reinforcement learning from AI feedback (RLAIF) trains a reward model using preferences generated by an AI labeler. In experiments on summarization and dialogue tasks, RLAIF produced results comparable to the studied RLHF baselines.<ref name="Lee2024">{{cite conference | ||
|last1=Lee |first1=Harrison | |last1=Lee |first1=Harrison | ||
|last2=Phatale |first2=Samrat | |last2=Phatale |first2=Samrat | ||
| Line 268: | Line 212: | ||
|display-authors=3 | |display-authors=3 | ||
|title=RLAIF vs. RLHF: Scaling Reinforcement Learning from Human Feedback with AI Feedback | |title=RLAIF vs. RLHF: Scaling Reinforcement Learning from Human Feedback with AI Feedback | ||
| | |book-title=Proceedings of the 41st International Conference on Machine Learning | ||
| | |series=Proceedings of Machine Learning Research | ||
|year= | |volume=235 | ||
}}</ref> | |pages=26874–26901 | ||
|year=2024 | |||
AI-generated feedback reduces dependence on human preference labels, but its results depend on the capabilities and prompts of the labeler model | |arxiv=2309.00267 | ||
}}</ref> AI-generated feedback reduces dependence on human preference labels, but its results depend on the capabilities and prompts of the labeler model and on the criteria used to guide its judgments.<ref name="Lee2024" /><ref name="Casper2023" /> | |||
=== Reasoning-focused post-training === | === Reasoning-focused post-training === | ||
Reasoning-oriented post-training uses examples or rewards that distinguish successful multi-step problem solving from plausible but incorrect output. Training signals can include worked solutions, model-generated rationales that pass a verifier, feedback on intermediate steps, or rewards based on a checkable final answer.<ref name="Kumar2025 | Reasoning-oriented post-training uses examples or rewards that distinguish successful multi-step problem solving from plausible but incorrect output. Training signals can include worked solutions, model-generated rationales that pass a verifier, feedback on intermediate steps, or rewards based on a checkable final answer.<ref name="Kumar2025" /> | ||
The Self-Taught Reasoner (STaR) iteratively generated rationales, retained rationales that led to correct answers, and fine-tuned on the retained examples.<ref name="Zelikman2022">{{cite | The Self-Taught Reasoner (STaR) iteratively generated rationales, retained rationales that led to correct answers, and fine-tuned on the retained examples.<ref name="Zelikman2022">{{cite conference | ||
|last1=Zelikman |first1=Eric | |last1=Zelikman |first1=Eric | ||
|last2=Wu |first2=Yuhuai | |last2=Wu |first2=Yuhuai | ||
|last3=Mu |first3=Jesse | |last3=Mu |first3=Jesse | ||
|last4=Goodman |first4=Noah D. | |last4=Goodman |first4=Noah D. | ||
|title=STaR: Bootstrapping Reasoning | |title=STaR: Bootstrapping Reasoning with Reasoning | ||
| | |book-title=Advances in Neural Information Processing Systems 35 | ||
|year=2022 | |year=2022 | ||
}}</ref> | |arxiv=2203.14465 | ||
}}</ref> Process supervision instead evaluates intermediate reasoning steps. In one study of mathematical reasoning, process-supervised reward models outperformed outcome-supervised reward models in the tested setting.<ref name="Lightman2023">{{cite arXiv | |||
|last1=Lightman |first1=Hunter | |||
|last1= | |last2=Kosaraju |first2=Vineet | ||
|last2= | |last3=Burda |first3=Yura | ||
|last3= | |last4=Edwards |first4=Harri | ||
|last4= | |||
|display-authors=3 | |display-authors=3 | ||
|title= | |title=Let's Verify Step by Step | ||
|eprint= | |eprint=2305.20050 | ||
|class=cs. | |class=cs.LG | ||
|year= | |year=2023 | ||
}}</ref> | }}</ref> Such results are specific to the studied tasks and models and do not establish that one reasoning-training recipe is uniformly preferable.<ref name="Kumar2025" /> | ||
=== Parameter-efficient adaptation === | === Parameter-efficient adaptation === | ||
Full fine-tuning updates all model parameters and can require substantial accelerator memory. Parameter-efficient methods instead train a smaller set of added or selected parameters while leaving most base-model weights fixed. Low-rank adaptation (LoRA) represents weight updates with trainable low-rank matrices, reducing the number of trainable parameters and the | Full fine-tuning updates all model parameters and can require substantial accelerator memory. Parameter-efficient methods instead train a smaller set of added or selected parameters while leaving most base-model weights fixed. Low-rank adaptation (LoRA) represents weight updates with trainable low-rank matrices, reducing the number of trainable parameters and the storage needed for separate task adaptations.<ref name="Hu2022">{{cite conference | ||
|last1=Hu |first1=Edward J. | |last1=Hu |first1=Edward J. | ||
|last2=Shen |first2=Yelong | |last2=Shen |first2=Yelong | ||
| Line 311: | Line 254: | ||
|display-authors=3 | |display-authors=3 | ||
|title=LoRA: Low-Rank Adaptation of Large Language Models | |title=LoRA: Low-Rank Adaptation of Large Language Models | ||
| | |book-title=International Conference on Learning Representations | ||
| | |year=2022 | ||
| | |arxiv=2106.09685 | ||
}}</ref> | }}</ref> | ||
Quantized low-rank adaptation (QLoRA) combines low-rank adapters with a quantized frozen base model. Its design backpropagates gradients through a frozen 4-bit quantized model into LoRA adapters, reducing memory use during fine-tuning.<ref name="Dettmers2023">{{cite | Quantized low-rank adaptation (QLoRA) combines low-rank adapters with a quantized frozen base model. Its design backpropagates gradients through a frozen 4-bit quantized model into LoRA adapters, reducing memory use during fine-tuning.<ref name="Dettmers2023">{{cite conference | ||
|last1=Dettmers |first1=Tim | |last1=Dettmers |first1=Tim | ||
|last2=Pagnoni |first2=Artidoro | |last2=Pagnoni |first2=Artidoro | ||
| Line 322: | Line 265: | ||
|last4=Zettlemoyer |first4=Luke | |last4=Zettlemoyer |first4=Luke | ||
|title=QLoRA: Efficient Finetuning of Quantized LLMs | |title=QLoRA: Efficient Finetuning of Quantized LLMs | ||
| | |book-title=Advances in Neural Information Processing Systems 36 | ||
|year=2023 | |year=2023 | ||
}}</ref> LoRA and QLoRA describe how an update is parameterized; they can be used with instruction | |arxiv=2305.14314 | ||
}}</ref> LoRA and QLoRA describe how an update is parameterized; they can be used with instruction tuning or preference optimization rather than constituting a separate behavioral objective.<ref name="Hu2022" /><ref name="Dettmers2023" /> | |||
=== Distillation and consolidation === | === Distillation and consolidation === | ||
[[Knowledge distillation]] trains a | [[Knowledge distillation]] trains a student model to reproduce selected behavior from a teacher model. In LLM post-training, the transferred signal can include token probabilities, generated responses, or instruction-following behavior. MiniLLM used an on-policy distillation objective based on reverse Kullback-Leibler divergence to train smaller generative language models from larger teachers.<ref name="Gu2024">{{cite conference | ||
|last1=Gu |first1=Yuxian | |last1=Gu |first1=Yuxian | ||
|last2=Dong |first2=Li | |last2=Dong |first2=Li | ||
| Line 335: | Line 278: | ||
|last4=Huang |first4=Minlie | |last4=Huang |first4=Minlie | ||
|title=MiniLLM: On-Policy Distillation of Large Language Models | |title=MiniLLM: On-Policy Distillation of Large Language Models | ||
| | |book-title=International Conference on Learning Representations | ||
|year=2024 | |||
|arxiv=2306.08543 | |||
}}</ref> Distillation can reduce deployment cost or consolidate behavior learned by a larger post-trained model, but the student remains dependent on the teacher outputs and on the coverage and filtering of the distillation data.<ref name="Gu2024" /><ref name="Tie2025" /> | |||
| | |||
| | |||
}}</ref> | |||
Distillation can | |||
== Data and evaluation == | == Data and evaluation == | ||
Post-training datasets differ by stage. SFT uses target responses; preference optimization uses comparisons or desirability labels; process supervision uses judgments on intermediate steps; and reinforcement learning | Post-training datasets differ by stage. SFT uses target responses; preference optimization uses comparisons or desirability labels; process supervision uses judgments on intermediate steps; and some reinforcement-learning settings use outcomes that can be checked by rules, tests, or answer matching.<ref name="Zhang2023" /><ref name="Kaufmann2025" /><ref name="Lightman2023" /> Synthetic data can expand coverage, but filtering is needed to remove invalid, repetitive, or low-quality generations.<ref name="Wang2023" /> | ||
Evaluation is normally matched to the intended effect of each stage. Instruction following can be assessed on held-out tasks and human judgments; | Evaluation is normally matched to the intended effect of each stage. Instruction following can be assessed on held-out tasks and human judgments; preference models can be compared on held-out comparisons; reasoning systems can be evaluated using answer correctness and, where available, judgments on intermediate steps; and safety-oriented post-training requires separate adversarial or policy-relevant tests.<ref name="Chung2024" /><ref name="Kaufmann2025" /><ref name="Lightman2023" /><ref name="Qi2023">{{cite arXiv | ||
|last1=Qi |first1=Xiangyu | |last1=Qi |first1=Xiangyu | ||
|last2=Zeng |first2=Yi | |last2=Zeng |first2=Yi | ||
| Line 366: | Line 297: | ||
|class=cs.CL | |class=cs.CL | ||
|year=2023 | |year=2023 | ||
}}</ref> Multi-stage | }}</ref> Multi-stage pipelines may use development benchmarks during training and separate unseen evaluations afterward; some also apply benchmark decontamination to reduce overlap between training and evaluation data.<ref name="Lambert2024" /> | ||
No single score measures all post-training goals. Improvements in instruction following, | No single score measures all post-training goals. Improvements in instruction following, preference fit, reasoning correctness, diversity, harmlessness, or domain performance can move differently under the same update.<ref name="Casper2023" /><ref name="Rafailov2024" /> Comparisons are also sensitive to the base model, training data, prompts, decoding settings, and evaluator, so results from one pipeline do not by themselves identify a universally preferable method.<ref name="Zhang2023" /><ref name="Kaufmann2025" /> | ||
== Limitations == | == Limitations == | ||
Post-training can improve targeted behavior while degrading other properties. | Post-training can improve targeted behavior while degrading other properties. Fine-tuning can weaken previously applied safety behavior; red-teaming experiments have shown safety regressions after fine-tuning with small adversarial datasets and, to a lesser degree, with some benign datasets in the tested models.<ref name="Qi2023" /> Sequential stages can also cause capability regressions or catastrophic forgetting, depending on the data and objectives used.<ref name="Tie2025" /><ref name="Kumar2025" /> | ||
Preference-based methods depend on imperfect feedback. Human labels can be inconsistent, unrepresentative, or difficult to obtain for tasks whose correctness is hard to judge, while AI-generated labels depend on the labeler model | Preference-based methods depend on imperfect feedback. Human labels can be inconsistent, unrepresentative, or difficult to obtain for tasks whose correctness is hard to judge, while AI-generated labels depend on the labeler model and its instructions.<ref name="Kaufmann2025" /><ref name="Casper2023" /><ref name="Lee2024" /> Optimization can also overfit a learned or implicit preference signal: performance measured by the training objective may continue to improve while quality under an external evaluator plateaus or deteriorates.<ref name="Rafailov2024" /> | ||
Multi-stage pipelines increase computational and engineering cost and make causal attribution difficult. A final behavior may depend on the base model, data curation, stage order, reward design, sampling, and evaluation choices.<ref name="Lambert2024" /><ref name="Tie2025" /> Limited disclosure of post-training datasets and recipes can hinder independent reproduction and comparison.<ref name="Lambert2024" /> | |||
== See also == | == See also == | ||