Archive:Training of large language models: Difference between revisions
→Long-context training: YaRN done |
|||
| Line 42: | Line 42: | ||
* <math>\gamma_k = 0</math> if the feature rotates by less than <math>\alpha</math> times over the length of a context: <math>\omega_k L < 2\pi \alpha</math>. | * <math>\gamma_k = 0</math> if the feature rotates by less than <math>\alpha</math> times over the length of a context: <math>\omega_k L < 2\pi \alpha</math>. | ||
* <math>\gamma_k</math> is linearly interpolated otherwise: <math>\gamma_k = \frac{\frac{\omega_k L}{2\pi}-\alpha}{\beta - \alpha}</math>. | * <math>\gamma_k</math> is linearly interpolated otherwise: <math>\gamma_k = \frac{\frac{\omega_k L}{2\pi}-\alpha}{\beta - \alpha}</math>. | ||
In YaRN, the logits in the softmax attention mechanism are scaled up. The idea is that stretching frequencies blurs the corresponding features, and scaling up the logits unblurs them. Empirically, the authors found <math>\operatorname{softmax}(z(1 + 0.1 \ln s)^2)</math> works well, where <math>z</math> are the logits. | In YaRN, the frequencies are scaled by ramp split with <math>\alpha = 1, \beta = 32</math>, and the logits in the softmax attention mechanism are scaled up. The idea is that stretching frequencies blurs the corresponding features, and scaling up the logits unblurs them. Empirically, the authors found <math>\operatorname{softmax}(z(1 + 0.1 \ln s)^2)</math> works well for context-extending Llama, where <math>z</math> are the logits. | ||
== Supervised fine-tuning == | == Supervised fine-tuning == | ||