Archive:Training of large language models: Difference between revisions

Reworked references to use final journal and conference publications where available
Reframed the scope conservatively and limited broad claims to reviews or explicit attribution
Line 1: Line 1:
{{Short description|Training methods for pretrained LLMs}}
{{Short description|Training methods used after LLM pretraining}}
{{Use American English|date=July 2026}}
{{Use American English|date=July 2026}}


'''Post-training of large language models''' is an umbrella term for training applied to a [[Large language model|large language model]] (LLM) after its initial large-scale pretraining. It starts from a pretrained or ''base'' model and changes its behavior for purposes such as following instructions, matching stated preferences, adapting to a domain, improving multi-step reasoning, or transferring behavior to a smaller model.<ref name="Tie2025">{{cite arXiv
'''Post-training of large language models''' is a term used in recent technical literature for training applied to a [[large language model]] (LLM) after its initial large-scale pretraining. The scope varies among surveys. Supervised fine-tuning or instruction tuning and preference-based alignment are common elements, while broader treatments differ on whether reasoning-focused training, parameter-efficient adaptation, distillation, tool integration, or inference-time scaling are included.<ref name="Tie2025">{{cite arXiv
  |last1=Tie |first1=Guiyao
  |last1=Tie |first1=Guiyao
  |last2=Zhao |first2=Zeli
  |last2=Zhao |first2=Zeli
Line 22: Line 22:
  |class=cs.CL
  |class=cs.CL
  |year=2025
  |year=2025
}}</ref> Common components include [[Fine-tuning (deep learning)|supervised fine-tuning]], instruction tuning, preference optimization, [[Reinforcement learning|reinforcement learning]], process or verifier-guided training, parameter-efficient adaptation, and [[Knowledge distillation|knowledge distillation]].<ref name="Zhang2023">{{cite arXiv
|last1=Zhang |first1=Shengyu
|last2=Dong |first2=Linfeng
|last3=Li |first3=Xiaoya
|last4=Zhang |first4=Sen
|display-authors=3
|title=Instruction Tuning for Large Language Models: A Survey
|eprint=2308.10792
|class=cs.CL
|year=2023
}}</ref><ref name="Kaufmann2025">{{cite journal
|last1=Kaufmann |first1=Timo
|last2=Weng |first2=Paul
|last3=Bengs |first3=Viktor
|last4=Hüllermeier |first4=Eyke
|title=A Survey of Reinforcement Learning from Human Feedback
|journal=Transactions on Machine Learning Research
|year=2025
|arxiv=2312.14925
}}</ref>
}}</ref>


Post-training data are usually more structured than pretraining corpora. Depending on the method, examples may contain instructions paired with target responses, comparisons between candidate responses, labels indicating desirable or undesirable output, model-generated critiques, intermediate reasoning steps, or outcomes that can be checked automatically.<ref name="Chung2024">{{cite journal
In training-centered usage, a pretrained or ''base'' model is further optimized with structured data such as instruction-response pairs, comparisons between candidate responses, step-level judgments, or outcomes that can be checked automatically. Practical systems may combine several stages, for example supervised training followed by preference optimization or reinforcement learning.<ref name="Chung2024">{{cite journal
  |last1=Chung |first1=Hyung Won
  |last1=Chung |first1=Hyung Won
  |last2=Hou |first2=Le
  |last2=Hou |first2=Le
Line 53: Line 34:
  |volume=25
  |volume=25
  |issue=70
  |issue=70
  |pages=1&ndash;53
  |pages=1-53
  |year=2024
  |year=2024
  |arxiv=2210.11416
  |arxiv=2210.11416
}}</ref><ref name="Ouyang2022">{{cite conference
}}</ref><ref name="Kaufmann2025">{{cite journal
  |last1=Ouyang |first1=Long
|last1=Kaufmann |first1=Timo
  |last2=Wu |first2=Jeff
|last2=Weng |first2=Paul
  |last3=Jiang |first3=Xu
|last3=Bengs |first3=Viktor
  |last4=Almeida |first4=Diogo
|last4=Hüllermeier |first4=Eyke
|title=A Survey of Reinforcement Learning from Human Feedback
|journal=Transactions on Machine Learning Research
|year=2025
|arxiv=2312.14925
}}</ref><ref name="Lightman2024">{{cite conference
  |last1=Lightman |first1=Hunter
  |last2=Kosaraju |first2=Vineet
  |last3=Burda |first3=Yura
  |last4=Edwards |first4=Harri
  |display-authors=3
  |display-authors=3
  |title=Training Language Models to Follow Instructions with Human Feedback
  |title=Let's Verify Step by Step
  |book-title=Advances in Neural Information Processing Systems 35
  |book-title=The Twelfth International Conference on Learning Representations
  |year=2022
  |year=2024
  |arxiv=2203.02155
  |arxiv=2305.20050
}}</ref> Practical systems often combine several stages rather than optimize one objective throughout training.<ref name="Lambert2024">{{cite arXiv
}}</ref><ref name="Lambert2024">{{cite arXiv
  |last1=Lambert |first1=Nathan
  |last1=Lambert |first1=Nathan
  |last2=Morrison |first2=Jacob
  |last2=Morrison |first2=Jacob
Line 80: Line 70:
== Scope and terminology ==
== Scope and terminology ==


The boundary of ''post-training'' varies across the literature. Recent surveys use it for overlapping sets of methods centered on adapting model behavior after pretraining, but they differ on whether to include topics such as multimodal adaptation, tool use, efficiency methods, and inference-time scaling.<ref name="Tie2025" /><ref name="Kumar2025" /> More established literatures treat instruction tuning and reinforcement learning from human feedback (RLHF) as distinct but frequently combined components of model adaptation.<ref name="Zhang2023" /><ref name="Kaufmann2025" /><ref name="Chung2024" />
Recent surveys use ''post-training'' as an umbrella label for several lines of research, including instruction tuning, preference alignment, reasoning-oriented training, efficiency methods, and distillation. Their boundaries differ, so the label does not imply a single objective or a universally agreed sequence of stages.<ref name="Tie2025" /><ref name="Kumar2025" />


The term is distinct from [[Large language model#Quantization|post-training quantization]], which reduces the numerical precision of an already trained network without referring to the broader process of teaching instruction following, preferences, or reasoning behavior.<ref name="Nagel2020">{{cite conference
Some surveys include increased computation at inference time in a broad account of post-training, while others center the term on additional training after pretraining.<ref name="Tie2025" /><ref name="Kumar2025" /> Post-training in this sense is also distinct from [[Large language model#Quantization|post-training quantization]], which lowers the numerical precision of an already trained network rather than teaching instruction following, preferences, or reasoning behavior.<ref name="Nagel2020">{{cite conference
  |last1=Nagel |first1=Markus
  |last1=Nagel |first1=Markus
  |last2=Amjad |first2=Rana Ali
  |last2=Amjad |first2=Rana Ali
Line 92: Line 82:
  |series=Proceedings of Machine Learning Research
  |series=Proceedings of Machine Learning Research
  |volume=119
  |volume=119
  |pages=7197&ndash;7206
  |pages=7197-7206
  |year=2020
  |year=2020
  |url=https://proceedings.mlr.press/v119/nagel20a.html
  |url=https://proceedings.mlr.press/v119/nagel20a.html
}}</ref>
}}</ref>
Post-training also does not imply a single objective. A pipeline may first establish basic instruction-following behavior with supervised examples, then train on preference data, and finally use reinforcement learning or distillation for a narrower capability or deployment target.<ref name="Ouyang2022" /><ref name="Lambert2024" />


== Development ==
== Development ==


Multitask instruction tuning was an early method later grouped within LLM post-training. T0 converted many supervised datasets into prompted tasks and trained a model to generalize to task types excluded from its training mixture.<ref name="Sanh2022">{{cite conference
The methods now described as post-training emerged through several lines of work.<ref name="Tie2025" /><ref name="Kumar2025" /> Multitask instruction tuning trained models on collections of tasks written as natural-language prompts. T0 converted supervised datasets into prompted tasks and evaluated generalization to tasks excluded from its training mixture.<ref name="Sanh2022">{{cite conference
  |last1=Sanh |first1=Victor
  |last1=Sanh |first1=Victor
  |last2=Webson |first2=Albert
  |last2=Webson |first2=Albert
Line 111: Line 99:
  |year=2022
  |year=2022
  |arxiv=2110.08207
  |arxiv=2110.08207
}}</ref> Later FLAN work scaled instruction tuning across model families, task collections, model sizes, and chain-of-thought data.<ref name="Chung2024" />
}}</ref> Later FLAN experiments studied instruction tuning across larger task mixtures, model families, model sizes, and chain-of-thought data.<ref name="Chung2024" />


Preference-based post-training developed alongside instruction tuning. Work on abstractive summarization trained a reward model from human comparisons and then optimized a language model against that reward.<ref name="Stiennon2020">{{cite conference
Preference-based training developed alongside instruction tuning. Work on abstractive summarization trained a reward model from human comparisons and optimized a language model against that reward.<ref name="Stiennon2020">{{cite conference
  |last1=Stiennon |first1=Nisan
  |last1=Stiennon |first1=Nisan
  |last2=Ouyang |first2=Long
  |last2=Ouyang |first2=Long
Line 123: Line 111:
  |year=2020
  |year=2020
  |arxiv=2009.01325
  |arxiv=2009.01325
}}</ref> InstructGPT used supervised fine-tuning on demonstrations, reward-model training on ranked outputs, and reinforcement learning from human feedback.<ref name="Ouyang2022" /> Reviews of RLHF describe the approach as combining human-feedback collection, reward learning, and policy optimization, while also identifying limitations in each stage.<ref name="Kaufmann2025" /><ref name="Casper2023">{{cite journal
}}</ref> InstructGPT used supervised fine-tuning on demonstrations, reward-model training on ranked outputs, and reinforcement learning from human feedback.<ref name="Ouyang2022">{{cite conference
|last1=Ouyang |first1=Long
|last2=Wu |first2=Jeff
|last3=Jiang |first3=Xu
|last4=Almeida |first4=Diogo
|display-authors=3
|title=Training Language Models to Follow Instructions with Human Feedback
|book-title=Advances in Neural Information Processing Systems 35
|year=2022
|arxiv=2203.02155
}}</ref> Reviews of reinforcement learning from human feedback describe the approach as a combination of feedback collection, reward learning, and policy optimization, with limitations at each stage.<ref name="Kaufmann2025" /><ref name="Casper2023">{{cite journal
  |last1=Casper |first1=Stephen
  |last1=Casper |first1=Stephen
  |last2=Davies |first2=Xander
  |last2=Davies |first2=Xander
Line 131: Line 129:
  |title=Open Problems and Fundamental Limitations of Reinforcement Learning from Human Feedback
  |title=Open Problems and Fundamental Limitations of Reinforcement Learning from Human Feedback
  |journal=Transactions on Machine Learning Research
  |journal=Transactions on Machine Learning Research
  |date=18 September 2023
  |year=2023
  |arxiv=2307.15217
  |arxiv=2307.15217
}}</ref>
}}</ref>


Direct preference optimization (DPO), introduced in 2023, reformulated a commonly used preference-learning objective as a classification loss, avoiding a separately trained reward model and an online policy-gradient stage.<ref name="Rafailov2023">{{cite conference
Direct preference optimization (DPO), introduced in 2023, reformulated a commonly used preference-learning objective as a classification loss and avoided the separately trained reward model and online policy-gradient stage used in a common reinforcement learning from human feedback pipeline.<ref name="Rafailov2023">{{cite conference
  |last1=Rafailov |first1=Rafael
  |last1=Rafailov |first1=Rafael
  |last2=Sharma |first2=Archit
  |last2=Sharma |first2=Archit
Line 145: Line 143:
  |year=2023
  |year=2023
  |arxiv=2305.18290
  |arxiv=2305.18290
}}</ref> Kahneman-Tversky optimization (KTO) was designed for data labeled desirable or undesirable without requiring a matched pair of responses.<ref name="Ethayarajh2024">{{cite conference
}}</ref> Kahneman-Tversky optimization (KTO) was subsequently proposed for data labeled desirable or undesirable without requiring matched response pairs.<ref name="Ethayarajh2024">{{cite conference
  |last1=Ethayarajh |first1=Kawin
  |last1=Ethayarajh |first1=Kawin
  |last2=Xu |first2=Winnie
  |last2=Xu |first2=Winnie
Line 163: Line 161:
=== Supervised fine-tuning and instruction tuning ===
=== Supervised fine-tuning and instruction tuning ===


In supervised fine-tuning (SFT), a model is trained to predict a target response for a given input. Instruction tuning is a form of SFT in which examples are organized as natural-language instructions paired with desired outputs. Research on instruction tuning examines task mixtures, instruction and response construction, model scale, data quality, and generalization to unseen tasks.<ref name="Zhang2023" /><ref name="Chung2024" />
In supervised fine-tuning (SFT), the model is trained to predict a target response for a given input. Instruction tuning is a form of SFT in which examples are organized as natural-language instructions paired with desired outputs. Research on instruction tuning examines task mixtures, model scale, response construction, data quality, and generalization to unseen tasks.<ref name="Chung2024" />


Instruction data can be written by people, converted from existing task datasets, generated synthetically, or assembled from several sources. Self-Instruct used a language model to generate instructions, inputs, and outputs, filtered invalid or near-duplicate examples, and then used the resulting data for fine-tuning.<ref name="Wang2023">{{cite conference
Instruction data can be written by people, converted from existing task datasets, generated synthetically, or assembled from several sources. Self-Instruct used a language model to generate instructions, inputs, and outputs, filtered invalid or near-duplicate examples, and then used the resulting data for fine-tuning.<ref name="Wang2023">{{cite conference
Line 173: Line 171:
  |title=Self-Instruct: Aligning Language Models with Self-Generated Instructions
  |title=Self-Instruct: Aligning Language Models with Self-Generated Instructions
  |book-title=Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics
  |book-title=Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics
|publisher=Association for Computational Linguistics
  |year=2023
  |year=2023
  |arxiv=2212.10560
  |arxiv=2212.10560
}}</ref> Results from particular instruction-tuning recipes do not imply that a fixed amount or composition of data is sufficient for every model or task; outcomes depend on the base model, task distribution, data construction, and training procedure.<ref name="Zhang2023" /><ref name="Chung2024" />
}}</ref> Results from individual recipes do not establish a fixed amount or composition of data that works for every model or task; outcomes vary with the base model, task distribution, and training procedure.<ref name="Chung2024" />


SFT is often the first stage of a longer pipeline. It can establish response formats and basic instruction following before preference optimization or reinforcement learning, and it can provide a starting policy that generates outputs for later comparison or verification.<ref name="Ouyang2022" /><ref name="Lambert2024" />
SFT often appears at the beginning of a multi-stage pipeline. It can establish response formats and basic instruction following before preference optimization or reinforcement learning, and it can provide a policy that generates outputs for later comparison or verification.<ref name="Ouyang2022" /><ref name="Lambert2024" />


=== Preference learning and alignment ===
=== Preference learning and alignment ===


Preference learning uses judgments about candidate outputs rather than only a single target answer. Feedback can be collected as rankings, pairwise comparisons, or labels such as desirable and undesirable. The training objective then increases the probability of preferred behavior while commonly constraining the updated model from moving too far from a reference policy.<ref name="Kaufmann2025" /><ref name="Ouyang2022" /><ref name="Rafailov2023" />
Preference learning uses judgments about candidate outputs rather than only a single target response. Feedback may be collected as rankings, pairwise comparisons, or labels such as desirable and undesirable. The model is then trained to increase the probability of preferred behavior, commonly subject to a constraint that limits divergence from a reference policy.<ref name="Kaufmann2025" /><ref name="Ouyang2022" /><ref name="Rafailov2023" />


==== Reward modeling and RLHF ====
==== Reward modeling and reinforcement learning from human feedback ====


A common RLHF pipeline begins with SFT, collects comparisons between model outputs, trains a reward model to predict those comparisons, and optimizes the language-model policy to increase the predicted reward.<ref name="Ouyang2022" /><ref name="Stiennon2020" /><ref name="Kaufmann2025" /> The reward model is a learned proxy for the collected feedback rather than a direct measurement of a universal human preference. The behavior learned by the policy therefore depends on the annotators, instructions, sampling process, and aggregation method used to create the feedback data.<ref name="Ouyang2022" /><ref name="Casper2023" />
A common reinforcement learning from human feedback (RLHF) pipeline begins with SFT, collects comparisons between model outputs, trains a reward model to predict those comparisons, and optimizes the language-model policy to increase the predicted reward.<ref name="Ouyang2022" /><ref name="Stiennon2020" /><ref name="Kaufmann2025" /> The reward model is a learned proxy for the collected feedback rather than a direct measurement of a universal human preference. The resulting behavior depends on the annotators, instructions, sampling process, and aggregation method used to construct the feedback data.<ref name="Ouyang2022" /><ref name="Casper2023" />


==== Direct preference optimization ====
==== Direct preference optimization ====


DPO learns directly from preferred and rejected responses. Its derivation expresses the optimal policy for a class of reward-regularized objectives in a form that can be fitted with a binary classification loss, eliminating the separately trained reward model and online reinforcement-learning loop used in a common RLHF pipeline.<ref name="Rafailov2023" /> DPO nevertheless depends on the coverage and quality of comparison data and on choices such as the reference policy and regularization strength.<ref name="Rafailov2023" /><ref name="Rafailov2024">{{cite conference
DPO learns directly from preferred and rejected responses. Its derivation expresses the optimal policy for a class of reward-regularized objectives in a form that can be fitted with a binary classification loss.<ref name="Rafailov2023" /> DPO still depends on the coverage and quality of comparison data and on choices such as the reference policy and regularization strength.<ref name="Rafailov2023" /><ref name="Rafailov2024">{{cite conference
  |last1=Rafailov |first1=Rafael
  |last1=Rafailov |first1=Rafael
  |last2=Chittepu |first2=Yaswanth
  |last2=Chittepu |first2=Yaswanth
Line 201: Line 200:
}}</ref>
}}</ref>


Related direct-alignment objectives use different assumptions about feedback. KTO, for example, was proposed for examples labeled desirable or undesirable without a matched alternative response.<ref name="Ethayarajh2024" /> These objectives are alternatives within preference-based post-training rather than evidence that one loss is uniformly best across datasets and applications.<ref name="Ethayarajh2024" /><ref name="Casper2023" />
Related objectives use different feedback assumptions. KTO, for example, was designed for examples labeled desirable or undesirable without a matched alternative response.<ref name="Ethayarajh2024" /> These methods are alternatives within preference-based training, not evidence that one objective is uniformly preferable across datasets and applications.<ref name="Ethayarajh2024" /><ref name="Casper2023" />


==== AI-generated feedback ====
==== AI-generated feedback ====


Feedback can be produced or assisted by another language model. Reinforcement learning from AI feedback (RLAIF) trains a reward model using preferences generated by an AI labeler. In experiments on summarization and dialogue tasks, RLAIF produced results comparable to the studied RLHF baselines.<ref name="Lee2024">{{cite conference
Feedback can also be generated or assisted by another language model. Reinforcement learning from AI feedback (RLAIF) trains a reward model using preferences produced by an AI labeler. In experiments on summarization and dialogue tasks, RLAIF produced results comparable to the studied RLHF baselines.<ref name="Lee2024">{{cite conference
  |last1=Lee |first1=Harrison
  |last1=Lee |first1=Harrison
  |last2=Phatale |first2=Samrat
  |last2=Phatale |first2=Samrat
Line 215: Line 214:
  |series=Proceedings of Machine Learning Research
  |series=Proceedings of Machine Learning Research
  |volume=235
  |volume=235
  |pages=26874&ndash;26901
  |pages=26874-26901
  |year=2024
  |year=2024
  |arxiv=2309.00267
  |arxiv=2309.00267
}}</ref> AI-generated feedback reduces dependence on human preference labels, but its results depend on the capabilities and prompts of the labeler model and on the criteria used to guide its judgments.<ref name="Lee2024" /><ref name="Casper2023" />
}}</ref> AI-generated feedback reduces dependence on human labels, but its results depend on the capabilities and prompts of the labeler model and on the criteria used to guide its judgments.<ref name="Lee2024" /><ref name="Casper2023" />


=== Reasoning-focused post-training ===
=== Reasoning-focused training ===


Reasoning-oriented post-training uses examples or rewards that distinguish successful multi-step problem solving from plausible but incorrect output. Training signals can include worked solutions, model-generated rationales that pass a verifier, feedback on intermediate steps, or rewards based on a checkable final answer.<ref name="Kumar2025" />
Some recent surveys include reasoning-focused methods within post-training.<ref name="Kumar2025" /><ref name="Tie2025" /> These methods use worked solutions, generated rationales, step-level feedback, or automatically checked outcomes to train multi-step problem solving.


The Self-Taught Reasoner (STaR) iteratively generated rationales, retained rationales that led to correct answers, and fine-tuned on the retained examples.<ref name="Zelikman2022">{{cite conference
The Self-Taught Reasoner (STaR) iteratively generated rationales, retained rationales that led to correct answers, and fine-tuned on the retained examples.<ref name="Zelikman2022">{{cite conference
Line 233: Line 232:
  |year=2022
  |year=2022
  |arxiv=2203.14465
  |arxiv=2203.14465
}}</ref> Process supervision instead evaluates intermediate reasoning steps. In one study of mathematical reasoning, process-supervised reward models outperformed outcome-supervised reward models in the tested setting.<ref name="Lightman2023">{{cite arXiv
}}</ref> Process supervision instead evaluates intermediate reasoning steps. In one study of mathematical reasoning, process-supervised reward models outperformed outcome-supervised reward models in the tested setting.<ref name="Lightman2024" /> Such results are specific to the studied tasks and models and do not establish that one reasoning-training recipe is uniformly preferable.<ref name="Kumar2025" />
|last1=Lightman |first1=Hunter
|last2=Kosaraju |first2=Vineet
|last3=Burda |first3=Yura
|last4=Edwards |first4=Harri
|display-authors=3
|title=Let's Verify Step by Step
|eprint=2305.20050
|class=cs.LG
|year=2023
}}</ref> Such results are specific to the studied tasks and models and do not establish that one reasoning-training recipe is uniformly preferable.<ref name="Kumar2025" />


=== Parameter-efficient adaptation ===
=== Parameter-efficient adaptation ===


Full fine-tuning updates all model parameters and can require substantial accelerator memory. Parameter-efficient methods instead train a smaller set of added or selected parameters while leaving most base-model weights fixed. Low-rank adaptation (LoRA) represents weight updates with trainable low-rank matrices, reducing the number of trainable parameters and the storage needed for separate task adaptations.<ref name="Hu2022">{{cite conference
Full fine-tuning updates all model parameters and can require substantial accelerator memory. Parameter-efficient methods train a smaller set of added or selected parameters while leaving most base-model weights fixed. Low-rank adaptation (LoRA) represents weight updates with trainable low-rank matrices, reducing the number of trainable parameters and the storage needed for separate task adaptations.<ref name="Hu2022">{{cite conference
  |last1=Hu |first1=Edward J.
  |last1=Hu |first1=Edward J.
  |last2=Shen |first2=Yelong
  |last2=Shen |first2=Yelong
Line 270: Line 259:
}}</ref> LoRA and QLoRA describe how an update is parameterized; they can be used with instruction tuning or preference optimization rather than constituting a separate behavioral objective.<ref name="Hu2022" /><ref name="Dettmers2023" />
}}</ref> LoRA and QLoRA describe how an update is parameterized; they can be used with instruction tuning or preference optimization rather than constituting a separate behavioral objective.<ref name="Hu2022" /><ref name="Dettmers2023" />


=== Distillation and consolidation ===
=== Distillation ===


[[Knowledge distillation]] trains a student model to reproduce selected behavior from a teacher model. In LLM post-training, the transferred signal can include token probabilities, generated responses, or instruction-following behavior. MiniLLM used an on-policy distillation objective based on reverse Kullback-Leibler divergence to train smaller generative language models from larger teachers.<ref name="Gu2024">{{cite conference
[[Knowledge distillation]] trains a student model to reproduce selected behavior from a teacher model. In LLM post-training, the transferred signal may include token probabilities, generated responses, or instruction-following behavior.<ref name="Tie2025" /> MiniLLM used an on-policy distillation objective based on reverse Kullback-Leibler divergence to train smaller generative language models from larger teachers.<ref name="Gu2024">{{cite conference
  |last1=Gu |first1=Yuxian
  |last1=Gu |first1=Yuxian
  |last2=Dong |first2=Li
  |last2=Dong |first2=Li
Line 278: Line 267:
  |last4=Huang |first4=Minlie
  |last4=Huang |first4=Minlie
  |title=MiniLLM: On-Policy Distillation of Large Language Models
  |title=MiniLLM: On-Policy Distillation of Large Language Models
  |book-title=International Conference on Learning Representations
  |book-title=The Twelfth International Conference on Learning Representations
  |year=2024
  |year=2024
  |arxiv=2306.08543
  |arxiv=2306.08543
}}</ref> Distillation can reduce deployment cost or consolidate behavior learned by a larger post-trained model, but the student remains dependent on the teacher outputs and on the coverage and filtering of the distillation data.<ref name="Gu2024" /><ref name="Tie2025" />
}}</ref> Distillation can reduce deployment cost or consolidate behavior learned by a larger model, but the student remains dependent on the teacher outputs and on the coverage and filtering of the distillation data.<ref name="Gu2024" /><ref name="Tie2025" />


== Data and evaluation ==
== Data and evaluation ==


Post-training datasets differ by stage. SFT uses target responses; preference optimization uses comparisons or desirability labels; process supervision uses judgments on intermediate steps; and some reinforcement-learning settings use outcomes that can be checked by rules, tests, or answer matching.<ref name="Zhang2023" /><ref name="Kaufmann2025" /><ref name="Lightman2023" /> Synthetic data can expand coverage, but filtering is needed to remove invalid, repetitive, or low-quality generations.<ref name="Wang2023" />
Post-training datasets differ by stage. SFT uses target responses, preference optimization uses comparisons or desirability labels, process supervision uses judgments on intermediate steps, and some reinforcement-learning settings use outcomes that can be checked by rules, tests, or answer matching.<ref name="Chung2024" /><ref name="Kaufmann2025" /><ref name="Lightman2024" /> Synthetic data can expand coverage, but filtering is used to remove invalid, repetitive, or low-quality generations.<ref name="Wang2023" />


Evaluation is normally matched to the intended effect of each stage. Instruction following can be assessed on held-out tasks and human judgments; preference models can be compared on held-out comparisons; reasoning systems can be evaluated using answer correctness and, where available, judgments on intermediate steps; and safety-oriented post-training requires separate adversarial or policy-relevant tests.<ref name="Chung2024" /><ref name="Kaufmann2025" /><ref name="Lightman2023" /><ref name="Qi2023">{{cite arXiv
Evaluation is matched to the intended effect of each stage. Instruction following can be assessed on held-out tasks and human judgments; preference models can be compared on held-out comparisons; reasoning systems can be evaluated using answer correctness and, where available, judgments on intermediate steps.<ref name="Chung2024" /><ref name="Kaufmann2025" /><ref name="Lightman2024" /> Multi-stage pipelines may use development benchmarks during training and separate unseen evaluations afterward; some also apply benchmark decontamination to reduce overlap between training and evaluation data.<ref name="Lambert2024" />
|last1=Qi |first1=Xiangyu
|last2=Zeng |first2=Yi
|last3=Xie |first3=Tinghao
|last4=Chen |first4=Pin-Yu
|display-authors=3
|title=Fine-tuning Aligned Language Models Compromises Safety, Even When Users Do Not Intend To!
|eprint=2310.03693
|class=cs.CL
|year=2023
}}</ref> Multi-stage pipelines may use development benchmarks during training and separate unseen evaluations afterward; some also apply benchmark decontamination to reduce overlap between training and evaluation data.<ref name="Lambert2024" />


No single score measures all post-training goals. Improvements in instruction following, preference fit, reasoning correctness, diversity, harmlessness, or domain performance can move differently under the same update.<ref name="Casper2023" /><ref name="Rafailov2024" /> Comparisons are also sensitive to the base model, training data, prompts, decoding settings, and evaluator, so results from one pipeline do not by themselves identify a universally preferable method.<ref name="Zhang2023" /><ref name="Kaufmann2025" />
No single score measures all post-training goals. Improvements in instruction following, preference fit, reasoning correctness, diversity, safety, or domain performance can move differently under the same update. Comparisons are also sensitive to the base model, training data, prompts, decoding settings, and evaluator.<ref name="Kaufmann2025" /><ref name="Casper2023" /><ref name="Rafailov2024" />


== Limitations ==
== Limitations ==


Post-training can improve targeted behavior while degrading other properties. Fine-tuning can weaken previously applied safety behavior; red-teaming experiments have shown safety regressions after fine-tuning with small adversarial datasets and, to a lesser degree, with some benign datasets in the tested models.<ref name="Qi2023" /> Sequential stages can also cause capability regressions or catastrophic forgetting, depending on the data and objectives used.<ref name="Tie2025" /><ref name="Kumar2025" />
Preference-based methods depend on imperfect feedback. Human labels can be inconsistent, unrepresentative, or difficult to obtain for tasks whose correctness is hard to judge, while AI-generated labels depend on the labeler model and its instructions.<ref name="Kaufmann2025" /><ref name="Casper2023" /><ref name="Lee2024" /> Optimization can also overfit a learned or implicit preference signal: performance measured by the training objective may continue to improve while quality under an external evaluator plateaus or deteriorates.<ref name="Rafailov2024" />


Preference-based methods depend on imperfect feedback. Human labels can be inconsistent, unrepresentative, or difficult to obtain for tasks whose correctness is hard to judge, while AI-generated labels depend on the labeler model and its instructions.<ref name="Kaufmann2025" /><ref name="Casper2023" /><ref name="Lee2024" /> Optimization can also overfit a learned or implicit preference signal: performance measured by the training objective may continue to improve while quality under an external evaluator plateaus or deteriorates.<ref name="Rafailov2024" />
Sequential stages can interfere with previously learned behavior. Recent post-training surveys discuss catastrophic forgetting, capability regressions, and trade-offs among alignment, diversity, and general task performance.<ref name="Tie2025" /><ref name="Kumar2025" /> These concerns depend on the base model, data, objective, and ordering of stages rather than following uniformly from every post-training method.


Multi-stage pipelines increase computational and engineering cost and make causal attribution difficult. A final behavior may depend on the base model, data curation, stage order, reward design, sampling, and evaluation choices.<ref name="Lambert2024" /><ref name="Tie2025" /> Limited disclosure of post-training datasets and recipes can hinder independent reproduction and comparison.<ref name="Lambert2024" />
Multi-stage pipelines increase computational and engineering cost and make causal attribution difficult. Final behavior may depend on the base model, data curation, stage order, reward design, sampling, and evaluation choices.<ref name="Lambert2024" /><ref name="Kaufmann2025" /> Limited disclosure of post-training datasets and recipes can hinder independent reproduction and comparison.<ref name="Lambert2024" />


== See also ==
== See also ==