Archive:Training of large language models: Difference between revisions

Reframed the scope conservatively and limited broad claims to reviews or explicit attribution
m 25 revisions imported: Import with full history
 
(21 intermediate revisions by 8 users not shown)
Line 1: Line 1:
{{Short description|Training methods used after LLM pretraining}}
{{Short description|Training methods used after LLM pretraining}}
{{Use American English|date=July 2026}}
{{Use American English|date=July 2026}}


'''Post-training of large language models''' is a term used in recent technical literature for training applied to a [[large language model]] (LLM) after its initial large-scale pretraining. The scope varies among surveys. Supervised fine-tuning or instruction tuning and preference-based alignment are common elements, while broader treatments differ on whether reasoning-focused training, parameter-efficient adaptation, distillation, tool integration, or inference-time scaling are included.<ref name="Tie2025">{{cite arXiv
A [[large language model]] (LLM) needs to be trained. The training of an LLM has become more complex over time.
|last1=Tie |first1=Guiyao
|last2=Zhao |first2=Zeli
|last3=Song |first3=Dingjie
|last4=Wei |first4=Fuyang
|display-authors=3
|title=A Survey on Post-training of Large Language Models
|eprint=2503.06072
|class=cs.CL
|year=2025
}}</ref><ref name="Kumar2025">{{cite arXiv
|last1=Kumar |first1=Komal
|last2=Ashraf |first2=Tajamul
|last3=Thawakar |first3=Omkar
|last4=Anwer |first4=Rao Muhammad
|display-authors=3
|title=LLM Post-Training: A Deep Dive into Reasoning Large Language Models
|eprint=2502.21321
|class=cs.CL
|year=2025
}}</ref>
 
In training-centered usage, a pretrained or ''base'' model is further optimized with structured data such as instruction-response pairs, comparisons between candidate responses, step-level judgments, or outcomes that can be checked automatically. Practical systems may combine several stages, for example supervised training followed by preference optimization or reinforcement learning.<ref name="Chung2024">{{cite journal
|last1=Chung |first1=Hyung Won
|last2=Hou |first2=Le
|last3=Longpre |first3=Shayne
|last4=Zoph |first4=Barret
|display-authors=3
|title=Scaling Instruction-Finetuned Language Models
|journal=Journal of Machine Learning Research
|volume=25
|issue=70
|pages=1-53
|year=2024
|arxiv=2210.11416
}}</ref><ref name="Kaufmann2025">{{cite journal
|last1=Kaufmann |first1=Timo
|last2=Weng |first2=Paul
|last3=Bengs |first3=Viktor
|last4=Hüllermeier |first4=Eyke
|title=A Survey of Reinforcement Learning from Human Feedback
|journal=Transactions on Machine Learning Research
|year=2025
|arxiv=2312.14925
}}</ref><ref name="Lightman2024">{{cite conference
|last1=Lightman |first1=Hunter
|last2=Kosaraju |first2=Vineet
|last3=Burda |first3=Yura
|last4=Edwards |first4=Harri
|display-authors=3
|title=Let's Verify Step by Step
|book-title=The Twelfth International Conference on Learning Representations
|year=2024
|arxiv=2305.20050
}}</ref><ref name="Lambert2024">{{cite arXiv
|last1=Lambert |first1=Nathan
|last2=Morrison |first2=Jacob
|last3=Pyatkin |first3=Valentina
|last4=Huang |first4=Shengyi
|display-authors=3
|title=Tulu 3: Pushing Frontiers in Open Language Model Post-Training
|eprint=2411.15124
|class=cs.CL
|year=2024
}}</ref>
 
== Scope and terminology ==
 
Recent surveys use ''post-training'' as an umbrella label for several lines of research, including instruction tuning, preference alignment, reasoning-oriented training, efficiency methods, and distillation. Their boundaries differ, so the label does not imply a single objective or a universally agreed sequence of stages.<ref name="Tie2025" /><ref name="Kumar2025" />
 
Some surveys include increased computation at inference time in a broad account of post-training, while others center the term on additional training after pretraining.<ref name="Tie2025" /><ref name="Kumar2025" /> Post-training in this sense is also distinct from [[Large language model#Quantization|post-training quantization]], which lowers the numerical precision of an already trained network rather than teaching instruction following, preferences, or reasoning behavior.<ref name="Nagel2020">{{cite conference
|last1=Nagel |first1=Markus
|last2=Amjad |first2=Rana Ali
|last3=van Baalen |first3=Mart
|last4=Louizos |first4=Christos
|display-authors=3
|title=Up or Down? Adaptive Rounding for Post-Training Quantization
|book-title=Proceedings of the 37th International Conference on Machine Learning
|series=Proceedings of Machine Learning Research
|volume=119
|pages=7197-7206
|year=2020
|url=https://proceedings.mlr.press/v119/nagel20a.html
}}</ref>
 
== Development ==
 
The methods now described as post-training emerged through several lines of work.<ref name="Tie2025" /><ref name="Kumar2025" /> Multitask instruction tuning trained models on collections of tasks written as natural-language prompts. T0 converted supervised datasets into prompted tasks and evaluated generalization to tasks excluded from its training mixture.<ref name="Sanh2022">{{cite conference
|last1=Sanh |first1=Victor
|last2=Webson |first2=Albert
|last3=Raffel |first3=Colin
|last4=Bach |first4=Stephen H.
|display-authors=3
|title=Multitask Prompted Training Enables Zero-Shot Task Generalization
|book-title=International Conference on Learning Representations
|year=2022
|arxiv=2110.08207
}}</ref> Later FLAN experiments studied instruction tuning across larger task mixtures, model families, model sizes, and chain-of-thought data.<ref name="Chung2024" />
 
Preference-based training developed alongside instruction tuning. Work on abstractive summarization trained a reward model from human comparisons and optimized a language model against that reward.<ref name="Stiennon2020">{{cite conference
|last1=Stiennon |first1=Nisan
|last2=Ouyang |first2=Long
|last3=Wu |first3=Jeff
|last4=Ziegler |first4=Daniel M.
|display-authors=3
|title=Learning to Summarize from Human Feedback
|book-title=Advances in Neural Information Processing Systems 33
|year=2020
|arxiv=2009.01325
}}</ref> InstructGPT used supervised fine-tuning on demonstrations, reward-model training on ranked outputs, and reinforcement learning from human feedback.<ref name="Ouyang2022">{{cite conference
|last1=Ouyang |first1=Long
|last2=Wu |first2=Jeff
|last3=Jiang |first3=Xu
|last4=Almeida |first4=Diogo
|display-authors=3
|title=Training Language Models to Follow Instructions with Human Feedback
|book-title=Advances in Neural Information Processing Systems 35
|year=2022
|arxiv=2203.02155
}}</ref> Reviews of reinforcement learning from human feedback describe the approach as a combination of feedback collection, reward learning, and policy optimization, with limitations at each stage.<ref name="Kaufmann2025" /><ref name="Casper2023">{{cite journal
|last1=Casper |first1=Stephen
|last2=Davies |first2=Xander
|last3=Shi |first3=Claudia
|last4=Gilbert |first4=Thomas Krendl
|display-authors=3
|title=Open Problems and Fundamental Limitations of Reinforcement Learning from Human Feedback
|journal=Transactions on Machine Learning Research
|year=2023
|arxiv=2307.15217
}}</ref>
 
Direct preference optimization (DPO), introduced in 2023, reformulated a commonly used preference-learning objective as a classification loss and avoided the separately trained reward model and online policy-gradient stage used in a common reinforcement learning from human feedback pipeline.<ref name="Rafailov2023">{{cite conference
|last1=Rafailov |first1=Rafael
|last2=Sharma |first2=Archit
|last3=Mitchell |first3=Eric
|last4=Ermon |first4=Stefano
|display-authors=3
|title=Direct Preference Optimization: Your Language Model Is Secretly a Reward Model
|book-title=Advances in Neural Information Processing Systems 36
|year=2023
|arxiv=2305.18290
}}</ref> Kahneman-Tversky optimization (KTO) was subsequently proposed for data labeled desirable or undesirable without requiring matched response pairs.<ref name="Ethayarajh2024">{{cite conference
|last1=Ethayarajh |first1=Kawin
|last2=Xu |first2=Winnie
|last3=Muennighoff |first3=Niklas
|last4=Jurafsky |first4=Dan
|display-authors=3
|title=KTO: Model Alignment as Prospect Theoretic Optimization
|book-title=Proceedings of the 41st International Conference on Machine Learning
|series=Proceedings of Machine Learning Research
|volume=235
|year=2024
|arxiv=2402.01306
}}</ref>
 
== Methods ==
 
=== Supervised fine-tuning and instruction tuning ===
 
In supervised fine-tuning (SFT), the model is trained to predict a target response for a given input. Instruction tuning is a form of SFT in which examples are organized as natural-language instructions paired with desired outputs. Research on instruction tuning examines task mixtures, model scale, response construction, data quality, and generalization to unseen tasks.<ref name="Chung2024" />
 
Instruction data can be written by people, converted from existing task datasets, generated synthetically, or assembled from several sources. Self-Instruct used a language model to generate instructions, inputs, and outputs, filtered invalid or near-duplicate examples, and then used the resulting data for fine-tuning.<ref name="Wang2023">{{cite conference
|last1=Wang |first1=Yizhong
|last2=Kordi |first2=Yeganeh
|last3=Mishra |first3=Swaroop
|last4=Liu |first4=Alisa
|display-authors=3
|title=Self-Instruct: Aligning Language Models with Self-Generated Instructions
|book-title=Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics
|publisher=Association for Computational Linguistics
|year=2023
|arxiv=2212.10560
}}</ref> Results from individual recipes do not establish a fixed amount or composition of data that works for every model or task; outcomes vary with the base model, task distribution, and training procedure.<ref name="Chung2024" />
 
SFT often appears at the beginning of a multi-stage pipeline. It can establish response formats and basic instruction following before preference optimization or reinforcement learning, and it can provide a policy that generates outputs for later comparison or verification.<ref name="Ouyang2022" /><ref name="Lambert2024" />
 
=== Preference learning and alignment ===
 
Preference learning uses judgments about candidate outputs rather than only a single target response. Feedback may be collected as rankings, pairwise comparisons, or labels such as desirable and undesirable. The model is then trained to increase the probability of preferred behavior, commonly subject to a constraint that limits divergence from a reference policy.<ref name="Kaufmann2025" /><ref name="Ouyang2022" /><ref name="Rafailov2023" />
 
==== Reward modeling and reinforcement learning from human feedback ====
 
A common reinforcement learning from human feedback (RLHF) pipeline begins with SFT, collects comparisons between model outputs, trains a reward model to predict those comparisons, and optimizes the language-model policy to increase the predicted reward.<ref name="Ouyang2022" /><ref name="Stiennon2020" /><ref name="Kaufmann2025" /> The reward model is a learned proxy for the collected feedback rather than a direct measurement of a universal human preference. The resulting behavior depends on the annotators, instructions, sampling process, and aggregation method used to construct the feedback data.<ref name="Ouyang2022" /><ref name="Casper2023" />
 
==== Direct preference optimization ====
 
DPO learns directly from preferred and rejected responses. Its derivation expresses the optimal policy for a class of reward-regularized objectives in a form that can be fitted with a binary classification loss.<ref name="Rafailov2023" /> DPO still depends on the coverage and quality of comparison data and on choices such as the reference policy and regularization strength.<ref name="Rafailov2023" /><ref name="Rafailov2024">{{cite conference
|last1=Rafailov |first1=Rafael
|last2=Chittepu |first2=Yaswanth
|last3=Park |first3=Ryan
|last4=Sikchi |first4=Harshit
|display-authors=3
|title=Scaling Laws for Reward Model Overoptimization in Direct Alignment Algorithms
|book-title=Advances in Neural Information Processing Systems 37
|year=2024
|arxiv=2406.02900
}}</ref>
 
Related objectives use different feedback assumptions. KTO, for example, was designed for examples labeled desirable or undesirable without a matched alternative response.<ref name="Ethayarajh2024" /> These methods are alternatives within preference-based training, not evidence that one objective is uniformly preferable across datasets and applications.<ref name="Ethayarajh2024" /><ref name="Casper2023" />
 
==== AI-generated feedback ====
 
Feedback can also be generated or assisted by another language model. Reinforcement learning from AI feedback (RLAIF) trains a reward model using preferences produced by an AI labeler. In experiments on summarization and dialogue tasks, RLAIF produced results comparable to the studied RLHF baselines.<ref name="Lee2024">{{cite conference
|last1=Lee |first1=Harrison
|last2=Phatale |first2=Samrat
|last3=Mansoor |first3=Hassan
|last4=Mesnard |first4=Thomas
|display-authors=3
|title=RLAIF vs. RLHF: Scaling Reinforcement Learning from Human Feedback with AI Feedback
|book-title=Proceedings of the 41st International Conference on Machine Learning
|series=Proceedings of Machine Learning Research
|volume=235
|pages=26874-26901
|year=2024
|arxiv=2309.00267
}}</ref> AI-generated feedback reduces dependence on human labels, but its results depend on the capabilities and prompts of the labeler model and on the criteria used to guide its judgments.<ref name="Lee2024" /><ref name="Casper2023" />
 
=== Reasoning-focused training ===
 
Some recent surveys include reasoning-focused methods within post-training.<ref name="Kumar2025" /><ref name="Tie2025" /> These methods use worked solutions, generated rationales, step-level feedback, or automatically checked outcomes to train multi-step problem solving.
 
The Self-Taught Reasoner (STaR) iteratively generated rationales, retained rationales that led to correct answers, and fine-tuned on the retained examples.<ref name="Zelikman2022">{{cite conference
|last1=Zelikman |first1=Eric
|last2=Wu |first2=Yuhuai
|last3=Mu |first3=Jesse
|last4=Goodman |first4=Noah D.
|title=STaR: Bootstrapping Reasoning with Reasoning
|book-title=Advances in Neural Information Processing Systems 35
|year=2022
|arxiv=2203.14465
}}</ref> Process supervision instead evaluates intermediate reasoning steps. In one study of mathematical reasoning, process-supervised reward models outperformed outcome-supervised reward models in the tested setting.<ref name="Lightman2024" /> Such results are specific to the studied tasks and models and do not establish that one reasoning-training recipe is uniformly preferable.<ref name="Kumar2025" />
 
=== Parameter-efficient adaptation ===
 
Full fine-tuning updates all model parameters and can require substantial accelerator memory. Parameter-efficient methods train a smaller set of added or selected parameters while leaving most base-model weights fixed. Low-rank adaptation (LoRA) represents weight updates with trainable low-rank matrices, reducing the number of trainable parameters and the storage needed for separate task adaptations.<ref name="Hu2022">{{cite conference
|last1=Hu |first1=Edward J.
|last2=Shen |first2=Yelong
|last3=Wallis |first3=Phillip
|last4=Allen-Zhu |first4=Zeyuan
|display-authors=3
|title=LoRA: Low-Rank Adaptation of Large Language Models
|book-title=International Conference on Learning Representations
|year=2022
|arxiv=2106.09685
}}</ref>
 
Quantized low-rank adaptation (QLoRA) combines low-rank adapters with a quantized frozen base model. Its design backpropagates gradients through a frozen 4-bit quantized model into LoRA adapters, reducing memory use during fine-tuning.<ref name="Dettmers2023">{{cite conference
|last1=Dettmers |first1=Tim
|last2=Pagnoni |first2=Artidoro
|last3=Holtzman |first3=Ari
|last4=Zettlemoyer |first4=Luke
|title=QLoRA: Efficient Finetuning of Quantized LLMs
|book-title=Advances in Neural Information Processing Systems 36
|year=2023
|arxiv=2305.14314
}}</ref> LoRA and QLoRA describe how an update is parameterized; they can be used with instruction tuning or preference optimization rather than constituting a separate behavioral objective.<ref name="Hu2022" /><ref name="Dettmers2023" />
 
=== Distillation ===


[[Knowledge distillation]] trains a student model to reproduce selected behavior from a teacher model. In LLM post-training, the transferred signal may include token probabilities, generated responses, or instruction-following behavior.<ref name="Tie2025" /> MiniLLM used an on-policy distillation objective based on reverse Kullback-Leibler divergence to train smaller generative language models from larger teachers.<ref name="Gu2024">{{cite conference
== Pretraining ==
|last1=Gu |first1=Yuxian
'''Pretraining''' is the training stage where the objective is next-token prediction.
|last2=Dong |first2=Li
|last3=Wei |first3=Furu
|last4=Huang |first4=Minlie
|title=MiniLLM: On-Policy Distillation of Large Language Models
|book-title=The Twelfth International Conference on Learning Representations
|year=2024
|arxiv=2306.08543
}}</ref> Distillation can reduce deployment cost or consolidate behavior learned by a larger model, but the student remains dependent on the teacher outputs and on the coverage and filtering of the distillation data.<ref name="Gu2024" /><ref name="Tie2025" />


== Data and evaluation ==
The pretraining stage can be split into several substages, one after another, as in [[curriculum learning]]. Each substage focuses on a different kind of learning, with later substages enabled by previous substages. All substages of pretraining would still use next-token prediction objective.


Post-training datasets differ by stage. SFT uses target responses, preference optimization uses comparisons or desirability labels, process supervision uses judgments on intermediate steps, and some reinforcement-learning settings use outcomes that can be checked by rules, tests, or answer matching.<ref name="Chung2024" /><ref name="Kaufmann2025" /><ref name="Lightman2024" /> Synthetic data can expand coverage, but filtering is used to remove invalid, repetitive, or low-quality generations.<ref name="Wang2023" />
=== Midtraining ===
'''Midtraining''' is a substage in pretraining. Like pretraining, the objective in midtraining is next-token prediction. They mainly differ in the design goals of the dataset that the model is predicting about. The pretraining dataset is designed to be larger and more diverse (such as [[Common Crawl]] or [[The Pile (dataset)|the Pile]]) to make the model generally capable, and the midtraining dataset is designed to be smaller, more specialized, and of higher quality, to efficiently make the model capable in specific areas.


Evaluation is matched to the intended effect of each stage. Instruction following can be assessed on held-out tasks and human judgments; preference models can be compared on held-out comparisons; reasoning systems can be evaluated using answer correctness and, where available, judgments on intermediate steps.<ref name="Chung2024" /><ref name="Kaufmann2025" /><ref name="Lightman2024" /> Multi-stage pipelines may use development benchmarks during training and separate unseen evaluations afterward; some also apply benchmark decontamination to reduce overlap between training and evaluation data.<ref name="Lambert2024" />
The term "mid-training" was first used by OpenAI in 2024,<ref>{{Cite arXiv |eprint=2506.20512 |class=cs.CL |first1=Zengzhi |last1=Wang |first2=Fan |last2=Zhou |title=OctoThinker: Mid-training Incentivizes Reinforcement Learning Scaling |date=2025-06-25 |last3=Li |first3=Xuefeng |last4=Liu |first4=Pengfei}}</ref> and came into more widespread use in 2025.<ref>{{Cite web |title=What's the deal with mid-training? {{!}} Vintage Data |url=https://vintagedata.org/blog/posts/what-is-mid-training |access-date=2026-10-19 |website=vintagedata.org}}</ref><ref>{{Cite arXiv |eprint=2501.00656 |class=cs.CL |last1=OLMo Team |title=2 OLMo 2 Furious |date=2025-10-08 }}</ref>


No single score measures all post-training goals. Improvements in instruction following, preference fit, reasoning correctness, diversity, safety, or domain performance can move differently under the same update. Comparisons are also sensitive to the base model, training data, prompts, decoding settings, and evaluator.<ref name="Kaufmann2025" /><ref name="Casper2023" /><ref name="Rafailov2024" />
=== Long-context training ===
{{Main|Context window#Context extension}}
'''Long-context training''' is a training substage within pretraining, designed to increase the effective context length of a model.<ref>{{Cite journal |last1=Gao |first1=Tianyu |last2=Wettig |first2=Alexander |last3=Yen |first3=Howard |last4=Chen |first4=Danqi |date=July 2025 |editor-last=Che |editor-first=Wanxiang |editor2-last=Nabende |editor2-first=Joyce |editor3-last=Shutova |editor3-first=Ekaterina |editor4-last=Pilehvar |editor4-first=Mohammad Taher |title=How to Train Long-Context Language Models (Effectively) |url=https://aclanthology.org/2025.acl-long.366/ |journal=Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) |location=Vienna, Austria |publisher=Association for Computational Linguistics |pages=7376–7399 |doi=10.18653/v1/2025.acl-long.366 |isbn=979-8-89176-251-0}}</ref> A language model can have a long context length, but unable to use it effectively, a phenomenon sometimes called "context decay". Consequently, it is the ''effective'' context length that is of interest, i.e. the length of the context that a model can use effectively.<ref>{{Cite arXiv |eprint=2404.06654 |class=cs.CL |first1=Cheng-Ping |last1=Hsieh |first2=Simeng |last2=Sun |title=RULER: What's the Real Context Size of Your Long-Context Language Models? |date=2024-08-06 |last3=Kriman |first3=Samuel |last4=Acharya |first4=Shantanu |last5=Rekesh |first5=Dima |last6=Jia |first6=Fei |last7=Zhang |first7=Yang |last8=Ginsburg |first8=Boris}}</ref>


== Limitations ==
A simple method of long-context training is simply to do next-token prediction on extended contexts, possibly extending more than once. Concretely, the model can be pretrained on context lengths up to 8K tokens, then up to 64K, then up to 512K. It was found that given a fixed training budget, training on a mixture of long contexts and short contexts resulted in higher performance than training on only long contexts.<ref>{{Cite journal |last1=Gao |first1=Tianyu |last2=Wettig |first2=Alexander |last3=Yen |first3=Howard |last4=Chen |first4=Danqi |date=July 2025 |editor-last=Che |editor-first=Wanxiang |editor2-last=Nabende |editor2-first=Joyce |editor3-last=Shutova |editor3-first=Ekaterina |editor4-last=Pilehvar |editor4-first=Mohammad Taher |title=How to Train Long-Context Language Models (Effectively) |url=https://aclanthology.org/2025.acl-long.366/ |journal=Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) |location=Vienna, Austria |publisher=Association for Computational Linguistics |pages=7376–7399 |doi=10.18653/v1/2025.acl-long.366 |isbn=979-8-89176-251-0}}</ref>


Preference-based methods depend on imperfect feedback. Human labels can be inconsistent, unrepresentative, or difficult to obtain for tasks whose correctness is hard to judge, while AI-generated labels depend on the labeler model and its instructions.<ref name="Kaufmann2025" /><ref name="Casper2023" /><ref name="Lee2024" /> Optimization can also overfit a learned or implicit preference signal: performance measured by the training objective may continue to improve while quality under an external evaluator plateaus or deteriorates.<ref name="Rafailov2024" />
Some architectures were designed specifically to enable efficient long-context training. An early example is [[Transformer (deep learning)#ALiBi|ALiBi (Attention with Linear Biases)]],<ref>{{Cite arXiv |eprint=2108.12409 |class=cs.CL |first1=Ofir |last1=Press |first2=Noah A. |last2=Smith |title=Train Short, Test Long: Attention with Linear Biases Enables Input Length Extrapolation |date=2022-04-22 |last3=Lewis |first3=Mike}}</ref> which had been replaced by various schemes for modifying the [[rotary positional embedding]] (RoPE) of a model. The modern representative is YaRN (Yet another [[Rotary positional embedding|RoPE]] extensioN) which has been used across many models, including [[DeepSeek (AI)|DeepSeek]], [[Qwen]], [[Kimi (AI)|Kimi]], and [[GPT-OSS]].<ref>{{Cite journal |last1=Peng |first1=Bowen |last2=Quesnelle |first2=Jeffrey |last3=Fan |first3=Honglu |last4=Shippole |first4=Enrico |date=2024-05-31 |title=YaRN: Efficient Context Window Extension of Large Language Models |url=https://proceedings.iclr.cc/paper_files/paper/2024/hash/874a4d89f2d04b4bcf9a2c19545cf040-Abstract-Conference.html |journal=International Conference on Learning Representations |language=en |volume=2024 |pages=31932–31951}}</ref>


Sequential stages can interfere with previously learned behavior. Recent post-training surveys discuss catastrophic forgetting, capability regressions, and trade-offs among alignment, diversity, and general task performance.<ref name="Tie2025" /><ref name="Kumar2025" /> These concerns depend on the base model, data, objective, and ordering of stages rather than following uniformly from every post-training method.
== Supervised fine-tuning ==
'''Supervised fine-tuning (SFT)''' is the stage where the objective is next-token prediction only on a subset of tokens. Like pretraining, considerations when designing and evaluating the effectiveness of SFT include data composition, context length, token efficiency, downstream performance, and so on. The data composition during the SFT stage critically affects the efficiency of SFT.<ref>{{Cite journal |last1=Xue |first1=Chao |last2=Wang |first2=Yao |last3=Liu |first3=Mengqiao |last4=Liang |first4=Di |last5=Han |first5=Xingsheng |last6=Liu |first6=Peiyang |last7=Wu |first7=Xianjie |last8=Lu |first8=Chenyao |last9=Jiang |first9=Lei |last10=Lu |first10=Yu |last11=Shi |first11=Haibo |last12=Liang |first12=Shuang |last13=Peng |first13=Minlong |last14=Salim |first14=Flora D. |date=July 2026 |editor-last=Liakata |editor-first=Maria |editor2-last=Moreira |editor2-first=Viviane P. |editor3-last=Zhang |editor3-first=Jiajun |editor4-last=Jurgens |editor4-first=David |title=Why Supervised Fine-Tuning Fails to Learn: A Systematic Study of Incomplete Learning in Large Language Models |url=https://aclanthology.org/2026.acl-long.1393/ |journal=Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) |location=San Diego, California, United States |publisher=Association for Computational Linguistics |pages=30186–30213 |doi=10.18653/v1/2026.acl-long.1393 |isbn=979-8-89176-390-6}}</ref><ref>{{Cite journal |last1=Dong |first1=Guanting |last2=Yuan |first2=Hongyi |last3=Lu |first3=Keming |last4=Li |first4=Chengpeng |last5=Xue |first5=Mingfeng |last6=Liu |first6=Dayiheng |last7=Wang |first7=Wei |last8=Yuan |first8=Zheng |last9=Zhou |first9=Chang |last10=Zhou |first10=Jingren |date=August 2024 |editor-last=Ku |editor-first=Lun-Wei |editor2-last=Martins |editor2-first=Andre |editor3-last=Srikumar |editor3-first=Vivek |title=How Abilities in Large Language Models are Affected by Supervised Fine-tuning Data Composition |url=https://aclanthology.org/2024.acl-long.12/ |journal=Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) |location=Bangkok, Thailand |publisher=Association for Computational Linguistics |pages=177–198 |doi=10.18653/v1/2024.acl-long.12}}</ref>


Multi-stage pipelines increase computational and engineering cost and make causal attribution difficult. Final behavior may depend on the base model, data curation, stage order, reward design, sampling, and evaluation choices.<ref name="Lambert2024" /><ref name="Kaufmann2025" /> Limited disclosure of post-training datasets and recipes can hinder independent reproduction and comparison.<ref name="Lambert2024" />
=== Instruction tuning ===
'''Instruction tuning''' is a type of SFT, in which the SFT dataset contains passages of the form <code><task instruction> <performing the task></code>. The objective is to predict only the tokens in the <code><performing the task></code> part.


== See also ==
== See also ==
Line 301: Line 42:


{{Reflist}}
{{Reflist}}
{{Artificial intelligence navbox}}


<!-- Categories to add after the move:
[[Category:Large language models]]
[[Category:Large language models]]
[[Category:Natural language processing]]
[[Category:Natural language processing]]
-->