Archive:Training of large language models: Difference between revisions

Line 35: Line 35:
In Positional Interpolation (PI),<ref>{{Cite arXiv |arxiv=2306.15595 |class=cs.CL |first1=Shouyuan |last1=Chen |first2=Sherman |last2=Wong |title=Extending Context Window of Large Language Models via Positional Interpolation |date=2023-06-28 |last3=Chen |first3=Liangjian |last4=Tian |first4=Yuandong}}</ref>  <math>\omega_k' = \omega_k/s</math>, stretching out each frequency equally. This was found to degrade performance beyond <math>s = 8</math>, as the model loses the high-frequency resolution that tells adjacent tokens apart.
In Positional Interpolation (PI),<ref>{{Cite arXiv |arxiv=2306.15595 |class=cs.CL |first1=Shouyuan |last1=Chen |first2=Sherman |last2=Wong |title=Extending Context Window of Large Language Models via Positional Interpolation |date=2023-06-28 |last3=Chen |first3=Liangjian |last4=Tian |first4=Yuandong}}</ref>  <math>\omega_k' = \omega_k/s</math>, stretching out each frequency equally. This was found to degrade performance beyond <math>s = 8</math>, as the model loses the high-frequency resolution that tells adjacent tokens apart.


In base change, <math>\omega_k' = (b')^{-\frac{k-1}{d}} </math> where <math>b' = bs^{\frac{d}{d-1}}</math>. Equivalently, <math>\omega_k' = \omega_k / s^{\frac{t-1}{d-1}}</math>. The effect is that the highest frequency is not stretched, the lowest frequency is stretched by <math>s</math>, and those in-between are interpolated.
In base change, <math>\omega_k' = (b')^{-\frac{k-1}{d}} </math> where <math>b' = bs^{\frac{d}{d-1}}</math>. Equivalently, <math>\omega_k' = \omega_k / s^{\frac{k-1}{d-1}}</math>. The effect is that the highest frequency is not stretched, the lowest frequency is stretched by <math>s</math>, and those in-between are interpolated.


In ramp split, high frequencies are not stretched, low frequencies are stretched by <math>s</math>, and those in-between are interpolated:<math display="block">\omega_k' = \gamma_k \omega_k + (1-\gamma_k) \omega_k / s</math>where
In ramp split, high frequencies are not stretched, low frequencies are stretched by <math>s</math>, and those in-between are interpolated:<math display="block">\omega_k' = \gamma_k \omega_k + (1-\gamma_k) \omega_k / s</math>where