Archive:Training of large language models: Difference between revisions
→Long-context training: YaRN |
→Long-context training: swap α β |
||
| Line 39: | Line 39: | ||
In ramp split, high frequencies are not stretched, low frequencies are stretched by <math>s</math>, and those in-between are interpolated:<math display="block">\omega_k' = \gamma_k \omega_k + (1-\gamma_k) \omega_k / s</math>where | In ramp split, high frequencies are not stretched, low frequencies are stretched by <math>s</math>, and those in-between are interpolated:<math display="block">\omega_k' = \gamma_k \omega_k + (1-\gamma_k) \omega_k / s</math>where | ||
* <math>\gamma_k = 1</math> if the feature rotates by more than <math>\ | * <math>\gamma_k = 1</math> if the feature rotates by more than <math>\beta</math> times over the length of a context: <math>\omega_k L > 2\pi \beta</math>. | ||
* <math>\gamma_k = 0</math> if the feature rotates by less than <math>\ | * <math>\gamma_k = 0</math> if the feature rotates by less than <math>\alpha</math> times over the length of a context: <math>\omega_k L < 2\pi \alpha</math>. | ||
* <math>\gamma_k</math> is linearly interpolated otherwise: <math>\gamma_k = \frac{\frac{\omega_k L}{2\pi}-\ | * <math>\gamma_k</math> is linearly interpolated otherwise: <math>\gamma_k = \frac{\frac{\omega_k L}{2\pi}-\alpha}{\beta - \alpha}</math>. | ||
== Supervised fine-tuning == | == Supervised fine-tuning == | ||
Revision as of 06:12, 26 October 2026
Template:Error:wrong namespace
| This article is being considered for deletion in accordance with Wikipedia's deletion policy. Please share your thoughts on the matter at this article's entry on the Articles for deletion page. Feel free to edit the article, but the article must not be blanked, and this notice must not be removed, until the discussion is closed. For more information, particularly on merging or moving the article during the discussion, read the Guide to deletion. |
A large language model (LLM) needs to be trained. The training of an LLM has become more complex over time.
Pretraining
Pretraining is the training stage where the objective is next-token prediction.
The pretraining stage can be split into several substages, one after another, as in curriculum learning. Each substage focuses on a different kind of learning, with later substages enabled by previous substages. All substages of pretraining would still use next-token prediction objective.
Midtraining
Midtraining is a substage in pretraining. Like pretraining, the objective in midtraining is next-token prediction. They mainly differ in the design goals of the dataset that the model is predicting about. The pretraining dataset is designed to be larger and more diverse (such as Common Crawl or the Pile) to make the model generally capable, and the midtraining dataset is designed to be smaller, more specialized, and of higher quality, to efficiently make the model capable in specific areas.
The term "mid-training" was first used by OpenAI in 2024,[1] and came into more widespread use in 2025.[2][3]
Long-context training
Long-context training is a training substage within pretraining, designed to increase the effective context length of a model.[4] A language model can have a long context length, but unable to use it effectively, a phenomenon sometimes called "context decay". Consequently, it is the effective context length that is of interest, i.e. the length of the context that a model can use effectively.[5]
A simple method of long-context training is simply to do next-token prediction on extended contexts, possibly extending more than once. Concretely, the model can be pretrained on context lengths up to 8K tokens, then up to 64K, then up to 512K. It was found that given a fixed training budget, training on a mixture of long contexts and short contexts resulted in higher performance than training on only long contexts.[6]
Some architectures were designed specifically to enable efficient long-context training. An early example is ALiBi (Attention with Linear Biases),[7] which had been replaced by various schemes for modifying the rotary positional embedding (RoPE) of a model. The modern representative is YaRN (Yet another RoPE extensioN) which has been used across many models, including DeepSeek, Qwen, Kimi, and GPT-OSS.[8]
In RoPE, the coordinates of a feature vector are paired: <math>v = (x_1, y_1, \dots, x_d, y_d) \in \R^{2d}</math>. More succinctly, we write it as a complex vector <math>v = (x_1 + iy_1, \dots, x_d + iy_d) \in \mathbb C^{d}</math>. Then, RoPE is defined by a sequence of angles <math>\omega_1, \dots, \omega_d \in [0, 2\pi)</math>. The <math>t</math>-th feature vector is represented by rotating its entries:<math display="block">(z_1, \dots, z_d) \mapsto (e^{i\omega_1 t} z_1, \dots, e^{i\omega_d t} z_d)</math>A standard RoPE angle setting is <math>\omega_k = b^{-\frac{k-1}{d}} </math> where <math>b = 10000</math>, common since the original RoPE paper and Llama.[9][10]
The key fact of RoPE is that the dot-product is translation-invariant. That is, given any feature vectors <math>v, w</math> and any integers <math>t, s, n</math>, we have <math>\operatorname{RoPE}(v, t) \cdot \operatorname{RoPE}(w, s) = \operatorname{RoPE}(v, t+n) \cdot \operatorname{RoPE}(w, s+n)</math>. The effect of RoPE is to allow the attention mechanism to apportion its attention by coordinate. For the coordinates with small <math>\omega</math>, RoPE implicitly states that these coordinates encode information that varies slowly over a sequence, and vice versa. Concretely, if <math>\omega_d \approx 1/10000</math>, then RoPE rotates <math>z_d</math> one cycle approximately every 63000 tokens, meaning that <math>z_d</math> is treated as encoding features that only vary over tens of thousands of tokens.
In general, context extension using RoPE consists of training a model on one context length <math>L</math> with one set of RoPE angles <math>\omega_1, \dots, \omega_d</math>, then its RoPE angles are modified to <math>\omega_1', \dots, \omega_d'</math>, and it is trained further on context length <math>L' = sL</math> for some <math>s>1</math>.
In Positional Interpolation (PI),[11] <math>\omega_k' = \omega_k/s</math>, stretching out each frequency equally. This was found to degrade performance beyond <math>s = 8</math>, as the model loses the high-frequency resolution that tells adjacent tokens apart.
In base change, <math>\omega_k' = (b')^{-\frac{k-1}{d}} </math> where <math>b' = bs^{\frac{d}{d-1}}</math>. Equivalently, <math>\omega_k' = \omega_k / s^{\frac{t-1}{d-1}}</math>. The effect is that the highest frequency is not stretched, the lowest frequency is stretched by <math>s</math>, and those in-between are interpolated.
In ramp split, high frequencies are not stretched, low frequencies are stretched by <math>s</math>, and those in-between are interpolated:<math display="block">\omega_k' = \gamma_k \omega_k + (1-\gamma_k) \omega_k / s</math>where
- <math>\gamma_k = 1</math> if the feature rotates by more than <math>\beta</math> times over the length of a context: <math>\omega_k L > 2\pi \beta</math>.
- <math>\gamma_k = 0</math> if the feature rotates by less than <math>\alpha</math> times over the length of a context: <math>\omega_k L < 2\pi \alpha</math>.
- <math>\gamma_k</math> is linearly interpolated otherwise: <math>\gamma_k = \frac{\frac{\omega_k L}{2\pi}-\alpha}{\beta - \alpha}</math>.
Supervised fine-tuning
Supervised fine-tuning (SFT) is the stage where the objective is next-token prediction only on a subset of tokens. Like pretraining, considerations when designing and evaluating the effectiveness of SFT include data composition, context length, token efficiency, downstream performance, and so on. The data composition during the SFT stage critically affects the efficiency of SFT.[12][13]
Instruction tuning
Instruction tuning is a type of SFT, in which the SFT dataset contains passages of the form <task instruction> <performing the task>. The objective is to predict only the tokens in the <performing the task> part.
See also
- AI alignment
- Fine-tuning (deep learning)
- Knowledge distillation
- Large language model
- LoRA (machine learning)
- Reasoning model
- Reinforcement learning from human feedback
References
- ↑ Template:Cite arXiv
- ↑ "What's the deal with mid-training? | Vintage Data". https://vintagedata.org/blog/posts/what-is-mid-training.
- ↑ Template:Cite arXiv
- ↑ Gao, Tianyu; Wettig, Alexander; Yen, Howard; Chen, Danqi (2025-07). Che, Wanxiang; Nabende, Joyce; Shutova, Ekaterina et al.. eds. "How to Train Long-Context Language Models (Effectively)". Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) (Vienna, Austria: Association for Computational Linguistics): 7376–7399. doi:. ISBN 979-8-89176-251-0. https://aclanthology.org/2025.acl-long.366/.
- ↑ Template:Cite arXiv
- ↑ Gao, Tianyu; Wettig, Alexander; Yen, Howard; Chen, Danqi (2025-07). Che, Wanxiang; Nabende, Joyce; Shutova, Ekaterina et al.. eds. "How to Train Long-Context Language Models (Effectively)". Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) (Vienna, Austria: Association for Computational Linguistics): 7376–7399. doi:. ISBN 979-8-89176-251-0. https://aclanthology.org/2025.acl-long.366/.
- ↑ Template:Cite arXiv
- ↑ Peng, Bowen; Quesnelle, Jeffrey; Fan, Honglu; Shippole, Enrico (2024-05-31). "YaRN: Efficient Context Window Extension of Large Language Models" (in en). International Conference on Learning Representations 2024: 31932–31951. https://proceedings.iclr.cc/paper_files/paper/2024/hash/874a4d89f2d04b4bcf9a2c19545cf040-Abstract-Conference.html.
- ↑ Su, Jianlin; Ahmed, Murtadha; Lu, Yu; Pan, Shengfeng; Bo, Wen; Liu, Yunfeng (2024-02). "RoFormer: Enhanced transformer with Rotary Position Embedding" (in en-US). Neurocomputing 568: 127063. doi:. ISSN 0925-2312. https://www.sciencedirect.com/science/article/abs/pii/S0925231223011864.
- ↑ Template:Cite arXiv
- ↑ Template:Cite arXiv
- ↑ Xue, Chao; Wang, Yao; Liu, Mengqiao; Liang, Di; Han, Xingsheng; Liu, Peiyang; Wu, Xianjie; Lu, Chenyao; et al. (2026-07). Liakata, Maria; Moreira, Viviane P.; Zhang, Jiajun et al.. eds. "Why Supervised Fine-Tuning Fails to Learn: A Systematic Study of Incomplete Learning in Large Language Models". Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) (San Diego, California, United States: Association for Computational Linguistics): 30186–30213. doi:. ISBN 979-8-89176-390-6. https://aclanthology.org/2026.acl-long.1393/.
- ↑ Dong, Guanting; Yuan, Hongyi; Lu, Keming; Li, Chengpeng; Xue, Mingfeng; Liu, Dayiheng; Wang, Wei; Yuan, Zheng; et al. (2024-08). Ku, Lun-Wei; Martins, Andre; Srikumar, Vivek. eds. "How Abilities in Large Language Models are Affected by Supervised Fine-tuning Data Composition". Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) (Bangkok, Thailand: Association for Computational Linguistics): 177–198. doi:. https://aclanthology.org/2024.acl-long.12/.