Archive:Training of large language models: Difference between revisions

No edit summary
m 25 revisions imported: Import with full history
 
(9 intermediate revisions by 3 users not shown)
Line 1: Line 1:
{{Short description|Training methods used after LLM pretraining}}
{{Short description|Training methods used after LLM pretraining}}
<!-- Please do not remove or change this AfD message until the discussion has been closed. -->
{{Article for deletion/dated|page=Training of large language models|timestamp=20261019075515|year=2026|month=October|day=19|substed=yes|help=off|outcome=draftification}}
<!-- Once discussion is closed, please place on talk page: {{Old AfD multi|page=Training of large language models|date=19 October 2026|result='''keep'''}} -->
<!-- End of AfD message, feel free to edit beyond this point -->


{{Use American English|date=July 2026}}
{{Use American English|date=July 2026}}
Line 18: Line 13:
'''Midtraining''' is a substage in pretraining. Like pretraining, the objective in midtraining is next-token prediction. They mainly differ in the design goals of the dataset that the model is predicting about. The pretraining dataset is designed to be larger and more diverse (such as [[Common Crawl]] or [[The Pile (dataset)|the Pile]]) to make the model generally capable, and the midtraining dataset is designed to be smaller, more specialized, and of higher quality, to efficiently make the model capable in specific areas.
'''Midtraining''' is a substage in pretraining. Like pretraining, the objective in midtraining is next-token prediction. They mainly differ in the design goals of the dataset that the model is predicting about. The pretraining dataset is designed to be larger and more diverse (such as [[Common Crawl]] or [[The Pile (dataset)|the Pile]]) to make the model generally capable, and the midtraining dataset is designed to be smaller, more specialized, and of higher quality, to efficiently make the model capable in specific areas.


The term "mid-training" was first used by OpenAI in 2024,<ref>{{Cite arXiv |arxiv=2506.20512 |class=cs.CL |first1=Zengzhi |last1=Wang |first2=Fan |last2=Zhou |title=OctoThinker: Mid-training Incentivizes Reinforcement Learning Scaling |date=2025-06-25 |last3=Li |first3=Xuefeng |last4=Liu |first4=Pengfei}}</ref> and came into more widespread use in 2025.<ref>{{Cite web |title=What's the deal with mid-training? {{!}} Vintage Data |url=https://vintagedata.org/blog/posts/what-is-mid-training |access-date=2026-10-19 |website=vintagedata.org}}</ref><ref>{{Cite arXiv |arxiv=2501.00656 |class=cs.CL |last1=OLMo Team |title=2 OLMo 2 Furious |date=2025-10-08 }}</ref>
The term "mid-training" was first used by OpenAI in 2024,<ref>{{Cite arXiv |eprint=2506.20512 |class=cs.CL |first1=Zengzhi |last1=Wang |first2=Fan |last2=Zhou |title=OctoThinker: Mid-training Incentivizes Reinforcement Learning Scaling |date=2025-06-25 |last3=Li |first3=Xuefeng |last4=Liu |first4=Pengfei}}</ref> and came into more widespread use in 2025.<ref>{{Cite web |title=What's the deal with mid-training? {{!}} Vintage Data |url=https://vintagedata.org/blog/posts/what-is-mid-training |access-date=2026-10-19 |website=vintagedata.org}}</ref><ref>{{Cite arXiv |eprint=2501.00656 |class=cs.CL |last1=OLMo Team |title=2 OLMo 2 Furious |date=2025-10-08 }}</ref>


=== Long-context training ===
=== Long-context training ===
'''Long-context training''' is a training substage within pretraining, designed to increase the effective context length of a model.<ref>{{Cite journal |last=Gao |first=Tianyu |last2=Wettig |first2=Alexander |last3=Yen |first3=Howard |last4=Chen |first4=Danqi |date=2025-07 |editor-last=Che |editor-first=Wanxiang |editor2-last=Nabende |editor2-first=Joyce |editor3-last=Shutova |editor3-first=Ekaterina |editor4-last=Pilehvar |editor4-first=Mohammad Taher |title=How to Train Long-Context Language Models (Effectively) |url=https://aclanthology.org/2025.acl-long.366/ |journal=Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) |location=Vienna, Austria |publisher=Association for Computational Linguistics |pages=7376–7399 |doi=10.18653/v1/2025.acl-long.366 |isbn=979-8-89176-251-0}}</ref> A language model can have a long context length, but unable to use it effectively, a phenomenon sometimes called "context decay". Consequently, it is the ''effective'' context length that is of interest, i.e. the length of the context that a model can use effectively.<ref>{{Cite arXiv |arxiv=2404.06654 |class=cs.CL |first1=Cheng-Ping |last1=Hsieh |first2=Simeng |last2=Sun |title=RULER: What's the Real Context Size of Your Long-Context Language Models? |date=2024-08-06 |last3=Kriman |first3=Samuel |last4=Acharya |first4=Shantanu |last5=Rekesh |first5=Dima |last6=Jia |first6=Fei |last7=Zhang |first7=Yang |last8=Ginsburg |first8=Boris}}</ref>
{{Main|Context window#Context extension}}
'''Long-context training''' is a training substage within pretraining, designed to increase the effective context length of a model.<ref>{{Cite journal |last1=Gao |first1=Tianyu |last2=Wettig |first2=Alexander |last3=Yen |first3=Howard |last4=Chen |first4=Danqi |date=July 2025 |editor-last=Che |editor-first=Wanxiang |editor2-last=Nabende |editor2-first=Joyce |editor3-last=Shutova |editor3-first=Ekaterina |editor4-last=Pilehvar |editor4-first=Mohammad Taher |title=How to Train Long-Context Language Models (Effectively) |url=https://aclanthology.org/2025.acl-long.366/ |journal=Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) |location=Vienna, Austria |publisher=Association for Computational Linguistics |pages=7376–7399 |doi=10.18653/v1/2025.acl-long.366 |isbn=979-8-89176-251-0}}</ref> A language model can have a long context length, but unable to use it effectively, a phenomenon sometimes called "context decay". Consequently, it is the ''effective'' context length that is of interest, i.e. the length of the context that a model can use effectively.<ref>{{Cite arXiv |eprint=2404.06654 |class=cs.CL |first1=Cheng-Ping |last1=Hsieh |first2=Simeng |last2=Sun |title=RULER: What's the Real Context Size of Your Long-Context Language Models? |date=2024-08-06 |last3=Kriman |first3=Samuel |last4=Acharya |first4=Shantanu |last5=Rekesh |first5=Dima |last6=Jia |first6=Fei |last7=Zhang |first7=Yang |last8=Ginsburg |first8=Boris}}</ref>
 
A simple method of long-context training is simply to do next-token prediction on extended contexts, possibly extending more than once. Concretely, the model can be pretrained on context lengths up to 8K tokens, then up to 64K, then up to 512K. It was found that given a fixed training budget, training on a mixture of long contexts and short contexts resulted in higher performance than training on only long contexts.<ref>{{Cite journal |last1=Gao |first1=Tianyu |last2=Wettig |first2=Alexander |last3=Yen |first3=Howard |last4=Chen |first4=Danqi |date=July 2025 |editor-last=Che |editor-first=Wanxiang |editor2-last=Nabende |editor2-first=Joyce |editor3-last=Shutova |editor3-first=Ekaterina |editor4-last=Pilehvar |editor4-first=Mohammad Taher |title=How to Train Long-Context Language Models (Effectively) |url=https://aclanthology.org/2025.acl-long.366/ |journal=Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) |location=Vienna, Austria |publisher=Association for Computational Linguistics |pages=7376–7399 |doi=10.18653/v1/2025.acl-long.366 |isbn=979-8-89176-251-0}}</ref>
 
Some architectures were designed specifically to enable efficient long-context training. An early example is [[Transformer (deep learning)#ALiBi|ALiBi (Attention with Linear Biases)]],<ref>{{Cite arXiv |eprint=2108.12409 |class=cs.CL |first1=Ofir |last1=Press |first2=Noah A. |last2=Smith |title=Train Short, Test Long: Attention with Linear Biases Enables Input Length Extrapolation |date=2022-04-22 |last3=Lewis |first3=Mike}}</ref> which had been replaced by various schemes for modifying the [[rotary positional embedding]] (RoPE) of a model. The modern representative is YaRN (Yet another [[Rotary positional embedding|RoPE]] extensioN) which has been used across many models, including [[DeepSeek (AI)|DeepSeek]], [[Qwen]], [[Kimi (AI)|Kimi]], and [[GPT-OSS]].<ref>{{Cite journal |last1=Peng |first1=Bowen |last2=Quesnelle |first2=Jeffrey |last3=Fan |first3=Honglu |last4=Shippole |first4=Enrico |date=2024-05-31 |title=YaRN: Efficient Context Window Extension of Large Language Models |url=https://proceedings.iclr.cc/paper_files/paper/2024/hash/874a4d89f2d04b4bcf9a2c19545cf040-Abstract-Conference.html |journal=International Conference on Learning Representations |language=en |volume=2024 |pages=31932–31951}}</ref>


A simple method of long-context training is simply to do next-token prediction on extended contexts, possibly extending more than once. Concretely, the model can be pretrained on context lengths up to 8K tokens, then up to 64K, then up to 512K. It was found that given a fixed training budget, training on a mixture of long contexts and short contexts resulted in higher performance than training on only long contexts.<ref>{{Cite journal |last=Gao |first=Tianyu |last2=Wettig |first2=Alexander |last3=Yen |first3=Howard |last4=Chen |first4=Danqi |date=2025-07 |editor-last=Che |editor-first=Wanxiang |editor2-last=Nabende |editor2-first=Joyce |editor3-last=Shutova |editor3-first=Ekaterina |editor4-last=Pilehvar |editor4-first=Mohammad Taher |title=How to Train Long-Context Language Models (Effectively) |url=https://aclanthology.org/2025.acl-long.366/ |journal=Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) |location=Vienna, Austria |publisher=Association for Computational Linguistics |pages=7376–7399 |doi=10.18653/v1/2025.acl-long.366 |isbn=979-8-89176-251-0}}</ref>
== Supervised fine-tuning ==
'''Supervised fine-tuning (SFT)''' is the stage where the objective is next-token prediction only on a subset of tokens. Like pretraining, considerations when designing and evaluating the effectiveness of SFT include data composition, context length, token efficiency, downstream performance, and so on. The data composition during the SFT stage critically affects the efficiency of SFT.<ref>{{Cite journal |last1=Xue |first1=Chao |last2=Wang |first2=Yao |last3=Liu |first3=Mengqiao |last4=Liang |first4=Di |last5=Han |first5=Xingsheng |last6=Liu |first6=Peiyang |last7=Wu |first7=Xianjie |last8=Lu |first8=Chenyao |last9=Jiang |first9=Lei |last10=Lu |first10=Yu |last11=Shi |first11=Haibo |last12=Liang |first12=Shuang |last13=Peng |first13=Minlong |last14=Salim |first14=Flora D. |date=July 2026 |editor-last=Liakata |editor-first=Maria |editor2-last=Moreira |editor2-first=Viviane P. |editor3-last=Zhang |editor3-first=Jiajun |editor4-last=Jurgens |editor4-first=David |title=Why Supervised Fine-Tuning Fails to Learn: A Systematic Study of Incomplete Learning in Large Language Models |url=https://aclanthology.org/2026.acl-long.1393/ |journal=Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) |location=San Diego, California, United States |publisher=Association for Computational Linguistics |pages=30186–30213 |doi=10.18653/v1/2026.acl-long.1393 |isbn=979-8-89176-390-6}}</ref><ref>{{Cite journal |last1=Dong |first1=Guanting |last2=Yuan |first2=Hongyi |last3=Lu |first3=Keming |last4=Li |first4=Chengpeng |last5=Xue |first5=Mingfeng |last6=Liu |first6=Dayiheng |last7=Wang |first7=Wei |last8=Yuan |first8=Zheng |last9=Zhou |first9=Chang |last10=Zhou |first10=Jingren |date=August 2024 |editor-last=Ku |editor-first=Lun-Wei |editor2-last=Martins |editor2-first=Andre |editor3-last=Srikumar |editor3-first=Vivek |title=How Abilities in Large Language Models are Affected by Supervised Fine-tuning Data Composition |url=https://aclanthology.org/2024.acl-long.12/ |journal=Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) |location=Bangkok, Thailand |publisher=Association for Computational Linguistics |pages=177–198 |doi=10.18653/v1/2024.acl-long.12}}</ref>


Some architectures were designed specifically to enable efficient long-context training. An early example is [[Transformer (deep learning)#ALiBi|ALiBi (Attention with Linear Biases)]].<ref>{{Cite arXiv |arxiv=2108.12409 |class=cs.CL |first1=Ofir |last1=Press |first2=Noah A. |last2=Smith |title=Train Short, Test Long: Attention with Linear Biases Enables Input Length Extrapolation |date=2022-04-22 |last3=Lewis |first3=Mike}}</ref> YaRN (Yet another RoPE extensioN) is a later method that has seen widespread use across many models, including DeepSeek, Qwen, Kimi, and [[GPT-OSS]].<ref>{{Cite journal |last=Peng |first=Bowen |last2=Quesnelle |first2=Jeffrey |last3=Fan |first3=Honglu |last4=Shippole |first4=Enrico |date=2024-05-31 |title=YaRN: Efficient Context Window Extension of Large Language Models |url=https://proceedings.iclr.cc/paper_files/paper/2024/hash/874a4d89f2d04b4bcf9a2c19545cf040-Abstract-Conference.html |journal=International Conference on Learning Representations |language=en |volume=2024 |pages=31932–31951}}</ref>
=== Instruction tuning ===
'''Instruction tuning''' is a type of SFT, in which the SFT dataset contains passages of the form <code><task instruction> <performing the task></code>. The objective is to predict only the tokens in the <code><performing the task></code> part.


== See also ==
== See also ==