Archive:Training of large language models: Difference between revisions
→Long-context training: YaRN |
→Long-context training: swap α β |
||
| Line 39: | Line 39: | ||
In ramp split, high frequencies are not stretched, low frequencies are stretched by <math>s</math>, and those in-between are interpolated:<math display="block">\omega_k' = \gamma_k \omega_k + (1-\gamma_k) \omega_k / s</math>where | In ramp split, high frequencies are not stretched, low frequencies are stretched by <math>s</math>, and those in-between are interpolated:<math display="block">\omega_k' = \gamma_k \omega_k + (1-\gamma_k) \omega_k / s</math>where | ||
* <math>\gamma_k = 1</math> if the feature rotates by more than <math>\ | * <math>\gamma_k = 1</math> if the feature rotates by more than <math>\beta</math> times over the length of a context: <math>\omega_k L > 2\pi \beta</math>. | ||
* <math>\gamma_k = 0</math> if the feature rotates by less than <math>\ | * <math>\gamma_k = 0</math> if the feature rotates by less than <math>\alpha</math> times over the length of a context: <math>\omega_k L < 2\pi \alpha</math>. | ||
* <math>\gamma_k</math> is linearly interpolated otherwise: <math>\gamma_k = \frac{\frac{\omega_k L}{2\pi}-\ | * <math>\gamma_k</math> is linearly interpolated otherwise: <math>\gamma_k = \frac{\frac{\omega_k L}{2\pi}-\alpha}{\beta - \alpha}</math>. | ||
== Supervised fine-tuning == | == Supervised fine-tuning == | ||