Archive:Training of large language models: Difference between revisions
→Long-context training: swap α β |
|||
| Line 35: | Line 35: | ||
In Positional Interpolation (PI),<ref>{{Cite arXiv |arxiv=2306.15595 |class=cs.CL |first1=Shouyuan |last1=Chen |first2=Sherman |last2=Wong |title=Extending Context Window of Large Language Models via Positional Interpolation |date=2023-06-28 |last3=Chen |first3=Liangjian |last4=Tian |first4=Yuandong}}</ref> <math>\omega_k' = \omega_k/s</math>, stretching out each frequency equally. This was found to degrade performance beyond <math>s = 8</math>, as the model loses the high-frequency resolution that tells adjacent tokens apart. | In Positional Interpolation (PI),<ref>{{Cite arXiv |arxiv=2306.15595 |class=cs.CL |first1=Shouyuan |last1=Chen |first2=Sherman |last2=Wong |title=Extending Context Window of Large Language Models via Positional Interpolation |date=2023-06-28 |last3=Chen |first3=Liangjian |last4=Tian |first4=Yuandong}}</ref> <math>\omega_k' = \omega_k/s</math>, stretching out each frequency equally. This was found to degrade performance beyond <math>s = 8</math>, as the model loses the high-frequency resolution that tells adjacent tokens apart. | ||
In base change, <math>\omega_k' = (b')^{-\frac{k-1}{d}} </math> where <math>b' = bs^{\frac{d}{d-1}}</math>. Equivalently, <math>\omega_k' = \omega_k / s^{\frac{ | In base change, <math>\omega_k' = (b')^{-\frac{k-1}{d}} </math> where <math>b' = bs^{\frac{d}{d-1}}</math>. Equivalently, <math>\omega_k' = \omega_k / s^{\frac{k-1}{d-1}}</math>. The effect is that the highest frequency is not stretched, the lowest frequency is stretched by <math>s</math>, and those in-between are interpolated. | ||
In ramp split, high frequencies are not stretched, low frequencies are stretched by <math>s</math>, and those in-between are interpolated:<math display="block">\omega_k' = \gamma_k \omega_k + (1-\gamma_k) \omega_k / s</math>where | In ramp split, high frequencies are not stretched, low frequencies are stretched by <math>s</math>, and those in-between are interpolated:<math display="block">\omega_k' = \gamma_k \omega_k + (1-\gamma_k) \omega_k / s</math>where | ||