Jump to content

Archive:Training of large language models: Difference between revisions

From IdeaWazaWiki
Remove tag
No edit summary
Line 18: Line 18:
'''Midtraining''' is a substage in pretraining. Like pretraining, the objective in midtraining is next-token prediction. They mainly differ in the design goals of the dataset that the model is predicting about. The pretraining dataset is designed to be larger and more diverse (such as [[Common Crawl]] or [[The Pile (dataset)|the Pile]]) to make the model generally capable, and the midtraining dataset is designed to be smaller, more specialized, and of higher quality, to efficiently make the model capable in specific areas.
'''Midtraining''' is a substage in pretraining. Like pretraining, the objective in midtraining is next-token prediction. They mainly differ in the design goals of the dataset that the model is predicting about. The pretraining dataset is designed to be larger and more diverse (such as [[Common Crawl]] or [[The Pile (dataset)|the Pile]]) to make the model generally capable, and the midtraining dataset is designed to be smaller, more specialized, and of higher quality, to efficiently make the model capable in specific areas.


The term "mid-training" was first used by OpenAI in 2024,<ref>{{Cite arXiv |arxiv=2506.20512 |class=cs.CL |first1=Zengzhi |last1=Wang |first2=Fan |last2=Zhou |title=OctoThinker: Mid-training Incentivizes Reinforcement Learning Scaling |date=2025-06-25 |last3=Li |first3=Xuefeng |last4=Liu |first4=Pengfei}}</ref> and came into more widespread use in 2025.<ref>{{Cite web |title=What's the deal with mid-training? {{!}} Vintage Data |url=https://vintagedata.org/blog/posts/what-is-mid-training |access-date=2026-10-19 |website=vintagedata.org}}</ref><ref>{{Cite arXiv |arxiv=2501.00656 |class=cs.CL |first1=Team |last1=OLMo |first2=Pete |last2=Walsh |title=2 OLMo 2 Furious |date=2025-10-08 |last3=Soldaini |first3=Luca |last4=Groeneveld |first4=Dirk |last5=Lo |first5=Kyle |last6=Arora |first6=Shane |last7=Bhagia |first7=Akshita |last8=Gu |first8=Yuling |last9=Huang |first9=Shengyi |last10=Jordan |first10=Matt |last11=Lambert |first11=Nathan |last12=Schwenk |first12=Dustin |last13=Tafjord |first13=Oyvind |last14=Anderson |first14=Taira |last15=Atkinson |first15=David |last16=Brahman |first16=Faeze |last17=Clark |first17=Christopher |last18=Dasigi |first18=Pradeep |last19=Dziri |first19=Nouha |last20=Ettinger |first20=Allyson |last21=Guerquin |first21=Michal |last22=Heineman |first22=David |last23=Ivison |first23=Hamish |last24=Koh |first24=Pang Wei |last25=Liu |first25=Jiacheng |last26=Malik |first26=Saumya |last27=Merrill |first27=William |last28=Miranda |first28=Lester James V. |last29=Morrison |first29=Jacob |last30=Murray |first30=Tyler |last31=Nam |first31=Crystal |last32=Poznanski |first32=Jake |last33=Pyatkin |first33=Valentina |last34=Rangapur |first34=Aman |last35=Schmitz |first35=Michael |last36=Skjonsberg |first36=Sam |last37=Wadden |first37=David |last38=Wilhelm |first38=Christopher |last39=Wilson |first39=Michael |last40=Zettlemoyer |first40=Luke |last41=Farhadi |first41=Ali |last42=Smith |first42=Noah A. |last43=Hajishirzi |first43=Hannaneh}}</ref>
The term "mid-training" was first used by OpenAI in 2024,<ref>{{Cite arXiv |arxiv=2506.20512 |class=cs.CL |first1=Zengzhi |last1=Wang |first2=Fan |last2=Zhou |title=OctoThinker: Mid-training Incentivizes Reinforcement Learning Scaling |date=2025-06-25 |last3=Li |first3=Xuefeng |last4=Liu |first4=Pengfei}}</ref> and came into more widespread use in 2025.<ref>{{Cite web |title=What's the deal with mid-training? {{!}} Vintage Data |url=https://vintagedata.org/blog/posts/what-is-mid-training |access-date=2026-10-19 |website=vintagedata.org}}</ref><ref>{{Cite arXiv |arxiv=2501.00656 |class=cs.CL |last1=OLMo Team |title=2 OLMo 2 Furious |date=2025-10-08 }}</ref>


=== Long-context training ===
=== Long-context training ===

Revision as of 06:49, 23 October 2026

Template:Short description

Template:Error:wrong namespace

Template:Use American English

A large language model (LLM) needs to be trained. The training of an LLM has become more complex over time.

Pretraining

Pretraining is the training stage where the objective is next-token prediction.

The pretraining stage can be split into several substages, one after another, as in curriculum learning. Each substage focuses on a different kind of learning, with later substages enabled by previous substages. All substages of pretraining would still use next-token prediction objective.

Midtraining

Midtraining is a substage in pretraining. Like pretraining, the objective in midtraining is next-token prediction. They mainly differ in the design goals of the dataset that the model is predicting about. The pretraining dataset is designed to be larger and more diverse (such as Common Crawl or the Pile) to make the model generally capable, and the midtraining dataset is designed to be smaller, more specialized, and of higher quality, to efficiently make the model capable in specific areas.

The term "mid-training" was first used by OpenAI in 2024,[1] and came into more widespread use in 2025.[2][3]

Long-context training

Long-context training is a training substage within pretraining, designed to increase the effective context length of a model.[4] A language model can have a long context length, but unable to use it effectively, a phenomenon sometimes called "context decay". Consequently, it is the effective context length that is of interest, i.e. the length of the context that a model can use effectively.[5]

A simple method of long-context training is simply to do next-token prediction on extended contexts, possibly extending more than once. Concretely, the model can be pretrained on context lengths up to 8K tokens, then up to 64K, then up to 512K. It was found that given a fixed training budget, training on a mixture of long contexts and short contexts resulted in higher performance than training on only long contexts.[6]

Some architectures were designed specifically to enable efficient long-context training. An early example is ALiBi (Attention with Linear Biases).[7] YaRN (Yet another RoPE extensioN) is a later method that has seen widespread use across many models, including DeepSeek, Qwen, Kimi, and GPT-OSS.[8]

See also

References

  1. ↑ Template:Cite arXiv
  2. ↑ "What's the deal with mid-training? | Vintage Data". https://vintagedata.org/blog/posts/what-is-mid-training. 
  3. ↑ Template:Cite arXiv
  4. ↑ Gao, Tianyu; Wettig, Alexander; Yen, Howard; Chen, Danqi (2025-07). Che, Wanxiang; Nabende, Joyce; Shutova, Ekaterina et al.. eds. "How to Train Long-Context Language Models (Effectively)". Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) (Vienna, Austria: Association for Computational Linguistics): 7376–7399. doi:10.18653/v1/2025.acl-long.366. ISBN 979-8-89176-251-0. https://aclanthology.org/2025.acl-long.366/. 
  5. ↑ Template:Cite arXiv
  6. ↑ Gao, Tianyu; Wettig, Alexander; Yen, Howard; Chen, Danqi (2025-07). Che, Wanxiang; Nabende, Joyce; Shutova, Ekaterina et al.. eds. "How to Train Long-Context Language Models (Effectively)". Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) (Vienna, Austria: Association for Computational Linguistics): 7376–7399. doi:10.18653/v1/2025.acl-long.366. ISBN 979-8-89176-251-0. https://aclanthology.org/2025.acl-long.366/. 
  7. ↑ Template:Cite arXiv
  8. ↑ Peng, Bowen; Quesnelle, Jeffrey; Fan, Honglu; Shippole, Enrico (2024-05-31). "YaRN: Efficient Context Window Extension of Large Language Models" (in en). International Conference on Learning Representations 2024: 31932–31951. https://proceedings.iclr.cc/paper_files/paper/2024/hash/874a4d89f2d04b4bcf9a2c19545cf040-Abstract-Conference.html. 

Template:Artificial intelligence navbox