Archive:Training of large language models: Difference between revisions

Line 39: Line 39:
In ramp split, high frequencies are not stretched, low frequencies are stretched by <math>s</math>, and those in-between are interpolated:<math display="block">\omega_k' = \gamma_k \omega_k + (1-\gamma_k) \omega_k / s</math>where
In ramp split, high frequencies are not stretched, low frequencies are stretched by <math>s</math>, and those in-between are interpolated:<math display="block">\omega_k' = \gamma_k \omega_k + (1-\gamma_k) \omega_k / s</math>where


* <math>\gamma_k = 1</math> if the feature rotates by more than <math>\alpha</math> times over the length of a context: <math>\omega_k L > 2\pi \alpha</math>.  
* <math>\gamma_k = 1</math> if the feature rotates by more than <math>\beta</math> times over the length of a context: <math>\omega_k L > 2\pi \beta</math>.  
* <math>\gamma_k = 0</math> if the feature rotates by less than <math>\beta</math> times over the length of a context: <math>\omega_k L < 2\pi \beta</math>.  
* <math>\gamma_k = 0</math> if the feature rotates by less than <math>\alpha</math> times over the length of a context: <math>\omega_k L < 2\pi \alpha</math>.  
* <math>\gamma_k</math> is linearly interpolated otherwise: <math>\gamma_k = \frac{\frac{\omega_k L}{2\pi}-\beta}{\alpha - \beta}</math>.
* <math>\gamma_k</math> is linearly interpolated otherwise: <math>\gamma_k = \frac{\frac{\omega_k L}{2\pi}-\alpha}{\beta - \alpha}</math>.


== Supervised fine-tuning ==
== Supervised fine-tuning ==