Archive:Training of large language models: Difference between revisions

a bit of SFT
Line 25: Line 25:
A simple method of long-context training is simply to do next-token prediction on extended contexts, possibly extending more than once. Concretely, the model can be pretrained on context lengths up to 8K tokens, then up to 64K, then up to 512K. It was found that given a fixed training budget, training on a mixture of long contexts and short contexts resulted in higher performance than training on only long contexts.<ref>{{Cite journal |last=Gao |first=Tianyu |last2=Wettig |first2=Alexander |last3=Yen |first3=Howard |last4=Chen |first4=Danqi |date=2025-07 |editor-last=Che |editor-first=Wanxiang |editor2-last=Nabende |editor2-first=Joyce |editor3-last=Shutova |editor3-first=Ekaterina |editor4-last=Pilehvar |editor4-first=Mohammad Taher |title=How to Train Long-Context Language Models (Effectively) |url=https://aclanthology.org/2025.acl-long.366/ |journal=Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) |location=Vienna, Austria |publisher=Association for Computational Linguistics |pages=7376–7399 |doi=10.18653/v1/2025.acl-long.366 |isbn=979-8-89176-251-0}}</ref>
A simple method of long-context training is simply to do next-token prediction on extended contexts, possibly extending more than once. Concretely, the model can be pretrained on context lengths up to 8K tokens, then up to 64K, then up to 512K. It was found that given a fixed training budget, training on a mixture of long contexts and short contexts resulted in higher performance than training on only long contexts.<ref>{{Cite journal |last=Gao |first=Tianyu |last2=Wettig |first2=Alexander |last3=Yen |first3=Howard |last4=Chen |first4=Danqi |date=2025-07 |editor-last=Che |editor-first=Wanxiang |editor2-last=Nabende |editor2-first=Joyce |editor3-last=Shutova |editor3-first=Ekaterina |editor4-last=Pilehvar |editor4-first=Mohammad Taher |title=How to Train Long-Context Language Models (Effectively) |url=https://aclanthology.org/2025.acl-long.366/ |journal=Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) |location=Vienna, Austria |publisher=Association for Computational Linguistics |pages=7376–7399 |doi=10.18653/v1/2025.acl-long.366 |isbn=979-8-89176-251-0}}</ref>


Some architectures were designed specifically to enable efficient long-context training. An early example is [[Transformer (deep learning)#ALiBi|ALiBi (Attention with Linear Biases)]].<ref>{{Cite arXiv |arxiv=2108.12409 |class=cs.CL |first1=Ofir |last1=Press |first2=Noah A. |last2=Smith |title=Train Short, Test Long: Attention with Linear Biases Enables Input Length Extrapolation |date=2022-04-22 |last3=Lewis |first3=Mike}}</ref> YaRN (Yet another RoPE extensioN) is a later method that has seen widespread use across many models, including DeepSeek, Qwen, Kimi, and [[GPT-OSS]].<ref>{{Cite journal |last=Peng |first=Bowen |last2=Quesnelle |first2=Jeffrey |last3=Fan |first3=Honglu |last4=Shippole |first4=Enrico |date=2024-05-31 |title=YaRN: Efficient Context Window Extension of Large Language Models |url=https://proceedings.iclr.cc/paper_files/paper/2024/hash/874a4d89f2d04b4bcf9a2c19545cf040-Abstract-Conference.html |journal=International Conference on Learning Representations |language=en |volume=2024 |pages=31932–31951}}</ref>
Some architectures were designed specifically to enable efficient long-context training. An early example is [[Transformer (deep learning)#ALiBi|ALiBi (Attention with Linear Biases)]],<ref>{{Cite arXiv |arxiv=2108.12409 |class=cs.CL |first1=Ofir |last1=Press |first2=Noah A. |last2=Smith |title=Train Short, Test Long: Attention with Linear Biases Enables Input Length Extrapolation |date=2022-04-22 |last3=Lewis |first3=Mike}}</ref> which had been replaced by various schemes for modifying the [[rotary positional embedding]] (RoPE) of a model. The modern representative is YaRN (Yet another [[Rotary positional embedding|RoPE]] extensioN) which has been used across many models, including [[DeepSeek (AI)|DeepSeek]], [[Qwen]], [[Kimi (AI)|Kimi]], and [[GPT-OSS]].<ref>{{Cite journal |last=Peng |first=Bowen |last2=Quesnelle |first2=Jeffrey |last3=Fan |first3=Honglu |last4=Shippole |first4=Enrico |date=2024-05-31 |title=YaRN: Efficient Context Window Extension of Large Language Models |url=https://proceedings.iclr.cc/paper_files/paper/2024/hash/874a4d89f2d04b4bcf9a2c19545cf040-Abstract-Conference.html |journal=International Conference on Learning Representations |language=en |volume=2024 |pages=31932–31951}}</ref>
 
In RoPE, the coordinates of a feature vector are paired: <math>v = (x_1, y_1, \dots, x_d, y_d) \in \R^{2d}</math>. More succinctly, we write it as a complex vector <math>v = (x_1 + iy_1, \dots, x_d + iy_d) \in \mathbb C^{d}</math>. Then, RoPE is defined by a sequence of angles <math>\omega_1, \dots, \omega_d \in [0, 2\pi)</math>. The <math>t</math>-th feature vector is represented by rotating its entries:<math display="block">(z_1, \dots, z_d) \mapsto (e^{i\omega_1 t} z_1, \dots, e^{i\omega_d t} z_d)</math>A standard RoPE angle setting is <math>\omega_k = b^{-\frac{k-1}{d}} </math> where <math>b = 10000</math>, common since the original RoPE paper and [[Llama (language model)|Llama]].<ref>{{Cite journal |last=Su |first=Jianlin |last2=Ahmed |first2=Murtadha |last3=Lu |first3=Yu |last4=Pan |first4=Shengfeng |last5=Bo |first5=Wen |last6=Liu |first6=Yunfeng |date=2024-02 |title=RoFormer: Enhanced transformer with Rotary Position Embedding |url=https://www.sciencedirect.com/science/article/abs/pii/S0925231223011864 |journal=Neurocomputing |language=en-US |volume=568 |pages=127063 |doi=10.1016/j.neucom.2023.127063 |issn=0925-2312 |archive-url=http://web.archive.org/web/20260115034533/https://www.sciencedirect.com/science/article/abs/pii/S0925231223011864 |archive-date=2026-01-15}}</ref><ref name="l1arxiv">{{cite arXiv |eprint=2302.13971 |class=cs.CL |first1=Hugo |last1=Touvron |first2=Thibaut |last2=Lavril |title=LLaMA: Open and Efficient Foundation Language Models |last3=Izacard |first3=Gautier |last4=Martinet |first4=Xavier |last5=Lachaux |first5=Marie-Anne |last6=Lacroix |first6=Timothée |last7=Rozière |first7=Baptiste |last8=Goyal |first8=Naman |last9=Hambro |first9=Eric |last10=Azhar |first10=Faisal |last11=Rodriguez |first11=Aurelien |last12=Joulin |first12=Armand |last13=Grave |first13=Edouard |last14=Lample |first14=Guillaume |year=2023}}</ref>
 
The key fact of RoPE is that the dot-product is translation-invariant. That is, given any feature vectors <math>v, w</math> and any integers <math>t, s, n</math>, we have <math>\operatorname{RoPE}(v, t) \cdot \operatorname{RoPE}(w, s) = \operatorname{RoPE}(v, t+n) \cdot \operatorname{RoPE}(w, s+n)</math>. The effect of RoPE is to allow the attention mechanism to apportion its attention by coordinate. For the coordinates with small <math>\omega</math>, RoPE implicitly states that these coordinates encode information that varies slowly over a sequence, and vice versa. Concretely, if <math>\omega_d \approx 1/10000</math>, then RoPE rotates <math>z_d</math> one cycle approximately every 63000 tokens, meaning that <math>z_d</math> is treated as encoding features that only vary over tens of thousands of tokens.
 
In general, context extension using RoPE consists of training a model on one context length <math>L</math> with one set of RoPE angles <math>\omega_1, \dots, \omega_d</math>, then its RoPE angles are modified to <math>\omega_1', \dots, \omega_d'</math>, and it is trained further on context length <math>L' = sL</math> for some <math>s>1</math>.
 
In Positional Interpolation (PI),<ref>{{Cite arXiv |arxiv=2306.15595 |class=cs.CL |first1=Shouyuan |last1=Chen |first2=Sherman |last2=Wong |title=Extending Context Window of Large Language Models via Positional Interpolation |date=2023-06-28 |last3=Chen |first3=Liangjian |last4=Tian |first4=Yuandong}}</ref>  <math>\omega_k' = \omega_k/s</math>, stretching out each frequency equally. This was found to degrade performance beyond <math>s = 8</math>, as the model loses the high-frequency resolution that tells adjacent tokens apart.
 
In base change, <math>\omega_k' = (b')^{-\frac{k-1}{d}} </math> where <math>b' = bs^{\frac{d}{d-1}}</math>. Equivalently, <math>\omega_k' = \omega_k / s^{\frac{t-1}{d-1}}</math>. The effect is that the highest frequency is not stretched, the lowest frequency is stretched by <math>s</math>, and those in-between are interpolated.
 
In ramp split, high frequencies are not stretched, low frequencies are stretched by <math>s</math>, and those in-between are interpolated:<math display="block">\omega_k' = \gamma_k \omega_k + (1-\gamma_k) \omega_k / s</math>where
 
* <math>\gamma_k = 1</math> if the feature rotates by more than <math>\alpha</math> times over the length of a context: <math>\omega_k L > 2\pi \alpha</math>.
* <math>\gamma_k = 0</math> if the feature rotates by less than <math>\beta</math> times over the length of a context: <math>\omega_k L < 2\pi \beta</math>.
* <math>\gamma_k</math> is linearly interpolated otherwise: <math>\gamma_k = \frac{\frac{\omega_k L}{2\pi}-\beta}{\alpha - \beta}</math>.


== Supervised fine-tuning ==
== Supervised fine-tuning ==