1. LORA: LOW-RANK ADAPTATION OF LARGE
LANGUAGE MODELS
2023.3.9
유용상
NLP 티타임

2. Introduction
• We take inspiration from Li et al. (2018a); Aghajanyan et al. (2020) which
show that the learned over-parametrized models in fact reside on a low
intrinsic dimension.
• We hypothesize that the change in weights during model adaptation also
has a low “intrinsic rank”, leading to our proposed Low-Rank Adaptation
(LoRA) approach.

3. Introduction : Adventages
• A pre-trained model can be shared and used to build many small LoRA modules for
different tasks. We can freeze the shared model and efficiently switch tasks by replacing
the matrices in reducing the storage requirement and task-switching overhead
significantly.
• LoRA makes training more efficient and lowers the hardware barrier to entry by up to 3
times when using adaptive optimizers. optimize the injected, much smaller low-rank
matrices.
• Our simple linear design allows us to merge the trainable matrices with the frozen
weights when deployed, introducing no inference latency compared to a fully fine-tuned
model, by construction.
• LoRA is orthogonal to many prior methods and can be combined with many of them,
such as prefix-tuning.

4. Problem Statement
175B
82B
178B
530B
280B

5. Problem Statement

6. Aren’t Existing Solutions Good Enough?

7. Low-Rank Parametrized Update Matrices
• Forward Pass
• Update
W0 + ∆W = W0 + BA
• Hypothesis :
가중치에 대한 update도 adaptation 중 intrinsic rank가 낮을
것이다.

8. Low-Rank Parametrized Update Matrices

9. Low-Rank Parametrized Update Matrices

10. Low-Rank Parametrized Update Matrices

11. Empirical Experiments

12. Empirical Experiments

13. Conclusion
• Large-scale model을 효율적으로 튜닝하는 LoRA 제안
• Adapter류의 기법과 다르게 inference latency가 발생하지 않음
• Prefix-tuning과 다르게 usable sequence length를 줄일 필요가 없음
• 가중치 업데이트 행렬이 low intrinsic rank를 가진다고 가정
• 논문에선 LM에 초점을 맞췄지만 이론적으로 모든 dense layer에 적용
가능

14. PEFT
Blog : https://huggingface.co/blog/peft
https://4n3mone.tistory.com/7
Code : https://github.com/huggingface/peft