|
|
| (22 intermediate revisions by 8 users not shown) |
| Line 1: |
Line 1: |
| {{Short description|Training methods for pretrained LLMs}} | | {{Short description|Training methods used after LLM pretraining}} |
| | |
| {{Use American English|date=July 2026}} | | {{Use American English|date=July 2026}} |
|
| |
|
| '''Post-training of large language models''' is an umbrella term for training applied to a [[Large language model|large language model]] (LLM) after its initial large-scale pretraining. It starts from a pretrained or ''base'' model and changes its behavior for purposes such as following instructions, matching stated preferences, adapting to a domain, improving multi-step reasoning, or transferring behavior to a smaller model.<ref name="Tie2025">{{cite arXiv
| | A [[large language model]] (LLM) needs to be trained. The training of an LLM has become more complex over time. |
| |last1=Tie |first1=Guiyao
| |
| |last2=Zhao |first2=Zeli
| |
| |last3=Song |first3=Dingjie
| |
| |last4=Wei |first4=Fuyang
| |
| |display-authors=3
| |
| |title=A Survey on Post-training of Large Language Models
| |
| |eprint=2503.06072
| |
| |class=cs.CL
| |
| |year=2025
| |
| }}</ref><ref name="Kumar2025">{{cite arXiv
| |
| |last1=Kumar |first1=Komal
| |
| |last2=Ashraf |first2=Tajamul
| |
| |last3=Thawakar |first3=Omkar
| |
| |last4=Anwer |first4=Rao Muhammad
| |
| |display-authors=3
| |
| |title=LLM Post-Training: A Deep Dive into Reasoning Large Language Models
| |
| |eprint=2502.21321
| |
| |class=cs.CL
| |
| |year=2025
| |
| }}</ref> Common components include [[Fine-tuning (deep learning)|supervised fine-tuning]], instruction tuning, preference optimization, [[Reinforcement learning|reinforcement learning]], process or verifier-guided training, parameter-efficient adaptation, and [[Knowledge distillation|knowledge distillation]].<ref name="Zhang2023">{{cite arXiv
| |
| |last1=Zhang |first1=Shengyu
| |
| |last2=Dong |first2=Linfeng
| |
| |last3=Li |first3=Xiaoya
| |
| |last4=Zhang |first4=Sen
| |
| |display-authors=3
| |
| |title=Instruction Tuning for Large Language Models: A Survey
| |
| |eprint=2308.10792
| |
| |class=cs.CL
| |
| |year=2023
| |
| }}</ref><ref name="Kaufmann2025">{{cite journal
| |
| |last1=Kaufmann |first1=Timo
| |
| |last2=Weng |first2=Paul
| |
| |last3=Bengs |first3=Viktor
| |
| |last4=Hüllermeier |first4=Eyke
| |
| |title=A Survey of Reinforcement Learning from Human Feedback
| |
| |journal=Transactions on Machine Learning Research
| |
| |year=2025
| |
| |arxiv=2312.14925
| |
| }}</ref>
| |
| | |
| Post-training data are usually more structured than pretraining corpora. Depending on the method, examples may contain instructions paired with target responses, comparisons between candidate responses, labels indicating desirable or undesirable output, model-generated critiques, intermediate reasoning steps, or outcomes that can be checked automatically.<ref name="Chung2024">{{cite journal
| |
| |last1=Chung |first1=Hyung Won
| |
| |last2=Hou |first2=Le
| |
| |last3=Longpre |first3=Shayne
| |
| |last4=Zoph |first4=Barret
| |
| |display-authors=3
| |
| |title=Scaling Instruction-Finetuned Language Models
| |
| |journal=Journal of Machine Learning Research
| |
| |volume=25
| |
| |issue=70
| |
| |pages=1–53
| |
| |year=2024
| |
| |arxiv=2210.11416
| |
| }}</ref><ref name="Ouyang2022">{{cite conference
| |
| |last1=Ouyang |first1=Long
| |
| |last2=Wu |first2=Jeff
| |
| |last3=Jiang |first3=Xu
| |
| |last4=Almeida |first4=Diogo
| |
| |display-authors=3
| |
| |title=Training Language Models to Follow Instructions with Human Feedback
| |
| |book-title=Advances in Neural Information Processing Systems 35
| |
| |year=2022
| |
| |arxiv=2203.02155
| |
| }}</ref> Practical systems often combine several stages rather than optimize one objective throughout training.<ref name="Lambert2024">{{cite arXiv
| |
| |last1=Lambert |first1=Nathan
| |
| |last2=Morrison |first2=Jacob
| |
| |last3=Pyatkin |first3=Valentina
| |
| |last4=Huang |first4=Shengyi
| |
| |display-authors=3
| |
| |title=Tulu 3: Pushing Frontiers in Open Language Model Post-Training
| |
| |eprint=2411.15124
| |
| |class=cs.CL
| |
| |year=2024
| |
| }}</ref>
| |
| | |
| == Scope and terminology ==
| |
| | |
| The boundary of ''post-training'' varies across the literature. Recent surveys use it for overlapping sets of methods centered on adapting model behavior after pretraining, but they differ on whether to include topics such as multimodal adaptation, tool use, efficiency methods, and inference-time scaling.<ref name="Tie2025" /><ref name="Kumar2025" /> More established literatures treat instruction tuning and reinforcement learning from human feedback (RLHF) as distinct but frequently combined components of model adaptation.<ref name="Zhang2023" /><ref name="Kaufmann2025" /><ref name="Chung2024" />
| |
| | |
| The term is distinct from [[Large language model#Quantization|post-training quantization]], which reduces the numerical precision of an already trained network without referring to the broader process of teaching instruction following, preferences, or reasoning behavior.<ref name="Nagel2020">{{cite conference | |
| |last1=Nagel |first1=Markus
| |
| |last2=Amjad |first2=Rana Ali
| |
| |last3=van Baalen |first3=Mart
| |
| |last4=Louizos |first4=Christos
| |
| |display-authors=3
| |
| |title=Up or Down? Adaptive Rounding for Post-Training Quantization
| |
| |book-title=Proceedings of the 37th International Conference on Machine Learning
| |
| |series=Proceedings of Machine Learning Research
| |
| |volume=119
| |
| |pages=7197–7206
| |
| |year=2020
| |
| |url=https://proceedings.mlr.press/v119/nagel20a.html
| |
| }}</ref>
| |
| | |
| Post-training also does not imply a single objective. A pipeline may first establish basic instruction-following behavior with supervised examples, then train on preference data, and finally use reinforcement learning or distillation for a narrower capability or deployment target.<ref name="Ouyang2022" /><ref name="Lambert2024" />
| |
| | |
| == Development ==
| |
| | |
| Multitask instruction tuning was an early method later grouped within LLM post-training. T0 converted many supervised datasets into prompted tasks and trained a model to generalize to task types excluded from its training mixture.<ref name="Sanh2022">{{cite conference
| |
| |last1=Sanh |first1=Victor
| |
| |last2=Webson |first2=Albert
| |
| |last3=Raffel |first3=Colin
| |
| |last4=Bach |first4=Stephen H.
| |
| |display-authors=3
| |
| |title=Multitask Prompted Training Enables Zero-Shot Task Generalization
| |
| |book-title=International Conference on Learning Representations
| |
| |year=2022
| |
| |arxiv=2110.08207
| |
| }}</ref> Later FLAN work scaled instruction tuning across model families, task collections, model sizes, and chain-of-thought data.<ref name="Chung2024" />
| |
| | |
| Preference-based post-training developed alongside instruction tuning. Work on abstractive summarization trained a reward model from human comparisons and then optimized a language model against that reward.<ref name="Stiennon2020">{{cite conference
| |
| |last1=Stiennon |first1=Nisan
| |
| |last2=Ouyang |first2=Long
| |
| |last3=Wu |first3=Jeff
| |
| |last4=Ziegler |first4=Daniel M.
| |
| |display-authors=3
| |
| |title=Learning to Summarize from Human Feedback
| |
| |book-title=Advances in Neural Information Processing Systems 33
| |
| |year=2020
| |
| |arxiv=2009.01325
| |
| }}</ref> InstructGPT used supervised fine-tuning on demonstrations, reward-model training on ranked outputs, and reinforcement learning from human feedback.<ref name="Ouyang2022" /> Reviews of RLHF describe the approach as combining human-feedback collection, reward learning, and policy optimization, while also identifying limitations in each stage.<ref name="Kaufmann2025" /><ref name="Casper2023">{{cite journal
| |
| |last1=Casper |first1=Stephen
| |
| |last2=Davies |first2=Xander
| |
| |last3=Shi |first3=Claudia
| |
| |last4=Gilbert |first4=Thomas Krendl
| |
| |display-authors=3
| |
| |title=Open Problems and Fundamental Limitations of Reinforcement Learning from Human Feedback
| |
| |journal=Transactions on Machine Learning Research
| |
| |date=18 September 2023
| |
| |arxiv=2307.15217
| |
| }}</ref>
| |
| | |
| Direct preference optimization (DPO), introduced in 2023, reformulated a commonly used preference-learning objective as a classification loss, avoiding a separately trained reward model and an online policy-gradient stage.<ref name="Rafailov2023">{{cite conference
| |
| |last1=Rafailov |first1=Rafael
| |
| |last2=Sharma |first2=Archit
| |
| |last3=Mitchell |first3=Eric
| |
| |last4=Ermon |first4=Stefano
| |
| |display-authors=3
| |
| |title=Direct Preference Optimization: Your Language Model Is Secretly a Reward Model
| |
| |book-title=Advances in Neural Information Processing Systems 36
| |
| |year=2023
| |
| |arxiv=2305.18290
| |
| }}</ref> Kahneman-Tversky optimization (KTO) was designed for data labeled desirable or undesirable without requiring a matched pair of responses.<ref name="Ethayarajh2024">{{cite conference
| |
| |last1=Ethayarajh |first1=Kawin
| |
| |last2=Xu |first2=Winnie
| |
| |last3=Muennighoff |first3=Niklas
| |
| |last4=Jurafsky |first4=Dan
| |
| |display-authors=3
| |
| |title=KTO: Model Alignment as Prospect Theoretic Optimization
| |
| |book-title=Proceedings of the 41st International Conference on Machine Learning
| |
| |series=Proceedings of Machine Learning Research
| |
| |volume=235
| |
| |year=2024
| |
| |arxiv=2402.01306
| |
| }}</ref>
| |
| | |
| == Methods ==
| |
| | |
| === Supervised fine-tuning and instruction tuning ===
| |
| | |
| In supervised fine-tuning (SFT), a model is trained to predict a target response for a given input. Instruction tuning is a form of SFT in which examples are organized as natural-language instructions paired with desired outputs. Research on instruction tuning examines task mixtures, instruction and response construction, model scale, data quality, and generalization to unseen tasks.<ref name="Zhang2023" /><ref name="Chung2024" />
| |
| | |
| Instruction data can be written by people, converted from existing task datasets, generated synthetically, or assembled from several sources. Self-Instruct used a language model to generate instructions, inputs, and outputs, filtered invalid or near-duplicate examples, and then used the resulting data for fine-tuning.<ref name="Wang2023">{{cite conference
| |
| |last1=Wang |first1=Yizhong
| |
| |last2=Kordi |first2=Yeganeh
| |
| |last3=Mishra |first3=Swaroop
| |
| |last4=Liu |first4=Alisa
| |
| |display-authors=3
| |
| |title=Self-Instruct: Aligning Language Models with Self-Generated Instructions
| |
| |book-title=Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics
| |
| |year=2023
| |
| |arxiv=2212.10560
| |
| }}</ref> Results from particular instruction-tuning recipes do not imply that a fixed amount or composition of data is sufficient for every model or task; outcomes depend on the base model, task distribution, data construction, and training procedure.<ref name="Zhang2023" /><ref name="Chung2024" />
| |
| | |
| SFT is often the first stage of a longer pipeline. It can establish response formats and basic instruction following before preference optimization or reinforcement learning, and it can provide a starting policy that generates outputs for later comparison or verification.<ref name="Ouyang2022" /><ref name="Lambert2024" />
| |
| | |
| === Preference learning and alignment ===
| |
| | |
| Preference learning uses judgments about candidate outputs rather than only a single target answer. Feedback can be collected as rankings, pairwise comparisons, or labels such as desirable and undesirable. The training objective then increases the probability of preferred behavior while commonly constraining the updated model from moving too far from a reference policy.<ref name="Kaufmann2025" /><ref name="Ouyang2022" /><ref name="Rafailov2023" />
| |
| | |
| ==== Reward modeling and RLHF ====
| |
| | |
| A common RLHF pipeline begins with SFT, collects comparisons between model outputs, trains a reward model to predict those comparisons, and optimizes the language-model policy to increase the predicted reward.<ref name="Ouyang2022" /><ref name="Stiennon2020" /><ref name="Kaufmann2025" /> The reward model is a learned proxy for the collected feedback rather than a direct measurement of a universal human preference. The behavior learned by the policy therefore depends on the annotators, instructions, sampling process, and aggregation method used to create the feedback data.<ref name="Ouyang2022" /><ref name="Casper2023" />
| |
| | |
| ==== Direct preference optimization ====
| |
| | |
| DPO learns directly from preferred and rejected responses. Its derivation expresses the optimal policy for a class of reward-regularized objectives in a form that can be fitted with a binary classification loss, eliminating the separately trained reward model and online reinforcement-learning loop used in a common RLHF pipeline.<ref name="Rafailov2023" /> DPO nevertheless depends on the coverage and quality of comparison data and on choices such as the reference policy and regularization strength.<ref name="Rafailov2023" /><ref name="Rafailov2024">{{cite conference
| |
| |last1=Rafailov |first1=Rafael
| |
| |last2=Chittepu |first2=Yaswanth
| |
| |last3=Park |first3=Ryan
| |
| |last4=Sikchi |first4=Harshit
| |
| |display-authors=3
| |
| |title=Scaling Laws for Reward Model Overoptimization in Direct Alignment Algorithms
| |
| |book-title=Advances in Neural Information Processing Systems 37
| |
| |year=2024
| |
| |arxiv=2406.02900
| |
| }}</ref>
| |
| | |
| Related direct-alignment objectives use different assumptions about feedback. KTO, for example, was proposed for examples labeled desirable or undesirable without a matched alternative response.<ref name="Ethayarajh2024" /> These objectives are alternatives within preference-based post-training rather than evidence that one loss is uniformly best across datasets and applications.<ref name="Ethayarajh2024" /><ref name="Casper2023" />
| |
| | |
| ==== AI-generated feedback ====
| |
| | |
| Feedback can be produced or assisted by another language model. Reinforcement learning from AI feedback (RLAIF) trains a reward model using preferences generated by an AI labeler. In experiments on summarization and dialogue tasks, RLAIF produced results comparable to the studied RLHF baselines.<ref name="Lee2024">{{cite conference
| |
| |last1=Lee |first1=Harrison
| |
| |last2=Phatale |first2=Samrat
| |
| |last3=Mansoor |first3=Hassan
| |
| |last4=Mesnard |first4=Thomas
| |
| |display-authors=3
| |
| |title=RLAIF vs. RLHF: Scaling Reinforcement Learning from Human Feedback with AI Feedback
| |
| |book-title=Proceedings of the 41st International Conference on Machine Learning
| |
| |series=Proceedings of Machine Learning Research
| |
| |volume=235
| |
| |pages=26874–26901
| |
| |year=2024
| |
| |arxiv=2309.00267
| |
| }}</ref> AI-generated feedback reduces dependence on human preference labels, but its results depend on the capabilities and prompts of the labeler model and on the criteria used to guide its judgments.<ref name="Lee2024" /><ref name="Casper2023" />
| |
| | |
| === Reasoning-focused post-training ===
| |
| | |
| Reasoning-oriented post-training uses examples or rewards that distinguish successful multi-step problem solving from plausible but incorrect output. Training signals can include worked solutions, model-generated rationales that pass a verifier, feedback on intermediate steps, or rewards based on a checkable final answer.<ref name="Kumar2025" />
| |
| | |
| The Self-Taught Reasoner (STaR) iteratively generated rationales, retained rationales that led to correct answers, and fine-tuned on the retained examples.<ref name="Zelikman2022">{{cite conference
| |
| |last1=Zelikman |first1=Eric
| |
| |last2=Wu |first2=Yuhuai
| |
| |last3=Mu |first3=Jesse
| |
| |last4=Goodman |first4=Noah D.
| |
| |title=STaR: Bootstrapping Reasoning with Reasoning
| |
| |book-title=Advances in Neural Information Processing Systems 35
| |
| |year=2022
| |
| |arxiv=2203.14465
| |
| }}</ref> Process supervision instead evaluates intermediate reasoning steps. In one study of mathematical reasoning, process-supervised reward models outperformed outcome-supervised reward models in the tested setting.<ref name="Lightman2023">{{cite arXiv
| |
| |last1=Lightman |first1=Hunter
| |
| |last2=Kosaraju |first2=Vineet
| |
| |last3=Burda |first3=Yura
| |
| |last4=Edwards |first4=Harri
| |
| |display-authors=3
| |
| |title=Let's Verify Step by Step
| |
| |eprint=2305.20050
| |
| |class=cs.LG
| |
| |year=2023
| |
| }}</ref> Such results are specific to the studied tasks and models and do not establish that one reasoning-training recipe is uniformly preferable.<ref name="Kumar2025" />
| |
| | |
| === Parameter-efficient adaptation ===
| |
| | |
| Full fine-tuning updates all model parameters and can require substantial accelerator memory. Parameter-efficient methods instead train a smaller set of added or selected parameters while leaving most base-model weights fixed. Low-rank adaptation (LoRA) represents weight updates with trainable low-rank matrices, reducing the number of trainable parameters and the storage needed for separate task adaptations.<ref name="Hu2022">{{cite conference
| |
| |last1=Hu |first1=Edward J.
| |
| |last2=Shen |first2=Yelong
| |
| |last3=Wallis |first3=Phillip
| |
| |last4=Allen-Zhu |first4=Zeyuan
| |
| |display-authors=3
| |
| |title=LoRA: Low-Rank Adaptation of Large Language Models
| |
| |book-title=International Conference on Learning Representations
| |
| |year=2022
| |
| |arxiv=2106.09685
| |
| }}</ref>
| |
| | |
| Quantized low-rank adaptation (QLoRA) combines low-rank adapters with a quantized frozen base model. Its design backpropagates gradients through a frozen 4-bit quantized model into LoRA adapters, reducing memory use during fine-tuning.<ref name="Dettmers2023">{{cite conference
| |
| |last1=Dettmers |first1=Tim
| |
| |last2=Pagnoni |first2=Artidoro
| |
| |last3=Holtzman |first3=Ari
| |
| |last4=Zettlemoyer |first4=Luke
| |
| |title=QLoRA: Efficient Finetuning of Quantized LLMs
| |
| |book-title=Advances in Neural Information Processing Systems 36
| |
| |year=2023
| |
| |arxiv=2305.14314
| |
| }}</ref> LoRA and QLoRA describe how an update is parameterized; they can be used with instruction tuning or preference optimization rather than constituting a separate behavioral objective.<ref name="Hu2022" /><ref name="Dettmers2023" />
| |
| | |
| === Distillation and consolidation ===
| |
|
| |
|
| [[Knowledge distillation]] trains a student model to reproduce selected behavior from a teacher model. In LLM post-training, the transferred signal can include token probabilities, generated responses, or instruction-following behavior. MiniLLM used an on-policy distillation objective based on reverse Kullback-Leibler divergence to train smaller generative language models from larger teachers.<ref name="Gu2024">{{cite conference
| | == Pretraining == |
| |last1=Gu |first1=Yuxian
| | '''Pretraining''' is the training stage where the objective is next-token prediction. |
| |last2=Dong |first2=Li
| |
| |last3=Wei |first3=Furu
| |
| |last4=Huang |first4=Minlie
| |
| |title=MiniLLM: On-Policy Distillation of Large Language Models
| |
| |book-title=International Conference on Learning Representations
| |
| |year=2024
| |
| |arxiv=2306.08543
| |
| }}</ref> Distillation can reduce deployment cost or consolidate behavior learned by a larger post-trained model, but the student remains dependent on the teacher outputs and on the coverage and filtering of the distillation data.<ref name="Gu2024" /><ref name="Tie2025" />
| |
|
| |
|
| == Data and evaluation ==
| | The pretraining stage can be split into several substages, one after another, as in [[curriculum learning]]. Each substage focuses on a different kind of learning, with later substages enabled by previous substages. All substages of pretraining would still use next-token prediction objective. |
|
| |
|
| Post-training datasets differ by stage. SFT uses target responses; preference optimization uses comparisons or desirability labels; process supervision uses judgments on intermediate steps; and some reinforcement-learning settings use outcomes that can be checked by rules, tests, or answer matching.<ref name="Zhang2023" /><ref name="Kaufmann2025" /><ref name="Lightman2023" /> Synthetic data can expand coverage, but filtering is needed to remove invalid, repetitive, or low-quality generations.<ref name="Wang2023" />
| | === Midtraining === |
| | '''Midtraining''' is a substage in pretraining. Like pretraining, the objective in midtraining is next-token prediction. They mainly differ in the design goals of the dataset that the model is predicting about. The pretraining dataset is designed to be larger and more diverse (such as [[Common Crawl]] or [[The Pile (dataset)|the Pile]]) to make the model generally capable, and the midtraining dataset is designed to be smaller, more specialized, and of higher quality, to efficiently make the model capable in specific areas. |
|
| |
|
| Evaluation is normally matched to the intended effect of each stage. Instruction following can be assessed on held-out tasks and human judgments; preference models can be compared on held-out comparisons; reasoning systems can be evaluated using answer correctness and, where available, judgments on intermediate steps; and safety-oriented post-training requires separate adversarial or policy-relevant tests.<ref name="Chung2024" /><ref name="Kaufmann2025" /><ref name="Lightman2023" /><ref name="Qi2023">{{cite arXiv
| | The term "mid-training" was first used by OpenAI in 2024,<ref>{{Cite arXiv |eprint=2506.20512 |class=cs.CL |first1=Zengzhi |last1=Wang |first2=Fan |last2=Zhou |title=OctoThinker: Mid-training Incentivizes Reinforcement Learning Scaling |date=2025-06-25 |last3=Li |first3=Xuefeng |last4=Liu |first4=Pengfei}}</ref> and came into more widespread use in 2025.<ref>{{Cite web |title=What's the deal with mid-training? {{!}} Vintage Data |url=https://vintagedata.org/blog/posts/what-is-mid-training |access-date=2026-10-19 |website=vintagedata.org}}</ref><ref>{{Cite arXiv |eprint=2501.00656 |class=cs.CL |last1=OLMo Team |title=2 OLMo 2 Furious |date=2025-10-08 }}</ref> |
| |last1=Qi |first1=Xiangyu
| |
| |last2=Zeng |first2=Yi
| |
| |last3=Xie |first3=Tinghao
| |
| |last4=Chen |first4=Pin-Yu
| |
| |display-authors=3
| |
| |title=Fine-tuning Aligned Language Models Compromises Safety, Even When Users Do Not Intend To!
| |
| |eprint=2310.03693
| |
| |class=cs.CL
| |
| |year=2023
| |
| }}</ref> Multi-stage pipelines may use development benchmarks during training and separate unseen evaluations afterward; some also apply benchmark decontamination to reduce overlap between training and evaluation data.<ref name="Lambert2024" /> | |
|
| |
|
| No single score measures all post-training goals. Improvements in instruction following, preference fit, reasoning correctness, diversity, harmlessness, or domain performance can move differently under the same update.<ref name="Casper2023" /><ref name="Rafailov2024" /> Comparisons are also sensitive to the base model, training data, prompts, decoding settings, and evaluator, so results from one pipeline do not by themselves identify a universally preferable method.<ref name="Zhang2023" /><ref name="Kaufmann2025" />
| | === Long-context training === |
| | {{Main|Context window#Context extension}} |
| | '''Long-context training''' is a training substage within pretraining, designed to increase the effective context length of a model.<ref>{{Cite journal |last1=Gao |first1=Tianyu |last2=Wettig |first2=Alexander |last3=Yen |first3=Howard |last4=Chen |first4=Danqi |date=July 2025 |editor-last=Che |editor-first=Wanxiang |editor2-last=Nabende |editor2-first=Joyce |editor3-last=Shutova |editor3-first=Ekaterina |editor4-last=Pilehvar |editor4-first=Mohammad Taher |title=How to Train Long-Context Language Models (Effectively) |url=https://aclanthology.org/2025.acl-long.366/ |journal=Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) |location=Vienna, Austria |publisher=Association for Computational Linguistics |pages=7376–7399 |doi=10.18653/v1/2025.acl-long.366 |isbn=979-8-89176-251-0}}</ref> A language model can have a long context length, but unable to use it effectively, a phenomenon sometimes called "context decay". Consequently, it is the ''effective'' context length that is of interest, i.e. the length of the context that a model can use effectively.<ref>{{Cite arXiv |eprint=2404.06654 |class=cs.CL |first1=Cheng-Ping |last1=Hsieh |first2=Simeng |last2=Sun |title=RULER: What's the Real Context Size of Your Long-Context Language Models? |date=2024-08-06 |last3=Kriman |first3=Samuel |last4=Acharya |first4=Shantanu |last5=Rekesh |first5=Dima |last6=Jia |first6=Fei |last7=Zhang |first7=Yang |last8=Ginsburg |first8=Boris}}</ref> |
|
| |
|
| == Limitations == | | A simple method of long-context training is simply to do next-token prediction on extended contexts, possibly extending more than once. Concretely, the model can be pretrained on context lengths up to 8K tokens, then up to 64K, then up to 512K. It was found that given a fixed training budget, training on a mixture of long contexts and short contexts resulted in higher performance than training on only long contexts.<ref>{{Cite journal |last1=Gao |first1=Tianyu |last2=Wettig |first2=Alexander |last3=Yen |first3=Howard |last4=Chen |first4=Danqi |date=July 2025 |editor-last=Che |editor-first=Wanxiang |editor2-last=Nabende |editor2-first=Joyce |editor3-last=Shutova |editor3-first=Ekaterina |editor4-last=Pilehvar |editor4-first=Mohammad Taher |title=How to Train Long-Context Language Models (Effectively) |url=https://aclanthology.org/2025.acl-long.366/ |journal=Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) |location=Vienna, Austria |publisher=Association for Computational Linguistics |pages=7376–7399 |doi=10.18653/v1/2025.acl-long.366 |isbn=979-8-89176-251-0}}</ref> |
|
| |
|
| Post-training can improve targeted behavior while degrading other properties. Fine-tuning can weaken previously applied safety behavior; red-teaming experiments have shown safety regressions after fine-tuning with small adversarial datasets and, to a lesser degree, with some benign datasets in the tested models.<ref name="Qi2023" /> Sequential stages can also cause capability regressions or catastrophic forgetting, depending on the data and objectives used.<ref name="Tie2025" /><ref name="Kumar2025" />
| | Some architectures were designed specifically to enable efficient long-context training. An early example is [[Transformer (deep learning)#ALiBi|ALiBi (Attention with Linear Biases)]],<ref>{{Cite arXiv |eprint=2108.12409 |class=cs.CL |first1=Ofir |last1=Press |first2=Noah A. |last2=Smith |title=Train Short, Test Long: Attention with Linear Biases Enables Input Length Extrapolation |date=2022-04-22 |last3=Lewis |first3=Mike}}</ref> which had been replaced by various schemes for modifying the [[rotary positional embedding]] (RoPE) of a model. The modern representative is YaRN (Yet another [[Rotary positional embedding|RoPE]] extensioN) which has been used across many models, including [[DeepSeek (AI)|DeepSeek]], [[Qwen]], [[Kimi (AI)|Kimi]], and [[GPT-OSS]].<ref>{{Cite journal |last1=Peng |first1=Bowen |last2=Quesnelle |first2=Jeffrey |last3=Fan |first3=Honglu |last4=Shippole |first4=Enrico |date=2024-05-31 |title=YaRN: Efficient Context Window Extension of Large Language Models |url=https://proceedings.iclr.cc/paper_files/paper/2024/hash/874a4d89f2d04b4bcf9a2c19545cf040-Abstract-Conference.html |journal=International Conference on Learning Representations |language=en |volume=2024 |pages=31932–31951}}</ref> |
|
| |
|
| Preference-based methods depend on imperfect feedback. Human labels can be inconsistent, unrepresentative, or difficult to obtain for tasks whose correctness is hard to judge, while AI-generated labels depend on the labeler model and its instructions.<ref name="Kaufmann2025" /><ref name="Casper2023" /><ref name="Lee2024" /> Optimization can also overfit a learned or implicit preference signal: performance measured by the training objective may continue to improve while quality under an external evaluator plateaus or deteriorates.<ref name="Rafailov2024" />
| | == Supervised fine-tuning == |
| | '''Supervised fine-tuning (SFT)''' is the stage where the objective is next-token prediction only on a subset of tokens. Like pretraining, considerations when designing and evaluating the effectiveness of SFT include data composition, context length, token efficiency, downstream performance, and so on. The data composition during the SFT stage critically affects the efficiency of SFT.<ref>{{Cite journal |last1=Xue |first1=Chao |last2=Wang |first2=Yao |last3=Liu |first3=Mengqiao |last4=Liang |first4=Di |last5=Han |first5=Xingsheng |last6=Liu |first6=Peiyang |last7=Wu |first7=Xianjie |last8=Lu |first8=Chenyao |last9=Jiang |first9=Lei |last10=Lu |first10=Yu |last11=Shi |first11=Haibo |last12=Liang |first12=Shuang |last13=Peng |first13=Minlong |last14=Salim |first14=Flora D. |date=July 2026 |editor-last=Liakata |editor-first=Maria |editor2-last=Moreira |editor2-first=Viviane P. |editor3-last=Zhang |editor3-first=Jiajun |editor4-last=Jurgens |editor4-first=David |title=Why Supervised Fine-Tuning Fails to Learn: A Systematic Study of Incomplete Learning in Large Language Models |url=https://aclanthology.org/2026.acl-long.1393/ |journal=Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) |location=San Diego, California, United States |publisher=Association for Computational Linguistics |pages=30186–30213 |doi=10.18653/v1/2026.acl-long.1393 |isbn=979-8-89176-390-6}}</ref><ref>{{Cite journal |last1=Dong |first1=Guanting |last2=Yuan |first2=Hongyi |last3=Lu |first3=Keming |last4=Li |first4=Chengpeng |last5=Xue |first5=Mingfeng |last6=Liu |first6=Dayiheng |last7=Wang |first7=Wei |last8=Yuan |first8=Zheng |last9=Zhou |first9=Chang |last10=Zhou |first10=Jingren |date=August 2024 |editor-last=Ku |editor-first=Lun-Wei |editor2-last=Martins |editor2-first=Andre |editor3-last=Srikumar |editor3-first=Vivek |title=How Abilities in Large Language Models are Affected by Supervised Fine-tuning Data Composition |url=https://aclanthology.org/2024.acl-long.12/ |journal=Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) |location=Bangkok, Thailand |publisher=Association for Computational Linguistics |pages=177–198 |doi=10.18653/v1/2024.acl-long.12}}</ref> |
|
| |
|
| Multi-stage pipelines increase computational and engineering cost and make causal attribution difficult. A final behavior may depend on the base model, data curation, stage order, reward design, sampling, and evaluation choices.<ref name="Lambert2024" /><ref name="Tie2025" /> Limited disclosure of post-training datasets and recipes can hinder independent reproduction and comparison.<ref name="Lambert2024" />
| | === Instruction tuning === |
| | '''Instruction tuning''' is a type of SFT, in which the SFT dataset contains passages of the form <code><task instruction> <performing the task></code>. The objective is to predict only the tokens in the <code><performing the task></code> part. |
|
| |
|
| == See also == | | == See also == |
| Line 322: |
Line 42: |
|
| |
|
| {{Reflist}} | | {{Reflist}} |
| | {{Artificial intelligence navbox}} |
|
| |
|
| <!-- Categories to add after the move:
| |
| [[Category:Large language models]] | | [[Category:Large language models]] |
| [[Category:Natural language processing]] | | [[Category:Natural language processing]] |
| -->
| |