| LoRA: Low-Rank Adaptation of Large Language Models |
Edward Hu et. al. |
2021-10-16 |
LoRA introduces a resource-efficient approach for adapting large pre-trained language models, such as GPT-3, to specific tasks without the heavy costs of traditional fine-tuning. It maintains model quality, minimizes inference latency, and facilitates quick task-switching. |
| Parameter-Efficient Fine-Tuning Methods for Pretrained Language Models: A Critical Review and Assessment |
Lingling Xu et. al. |
2023-12-19 |
This paper critically reviews Parameter Efficient Fine-Tuning (PEFT) methods for large pretrained language models (PLMs), highlighting their benefits in resource-limited settings. It assesses these methods’ performance, efficiency, and memory usage across tasks like natural language understanding, machine translation, and generation. |
| PaLM 2 Technical Report |
Rohan Anil et. al. |
2023-05-17 |
We introduce PaLM 2, a new state-of-the-art language model that has better multilingual and reasoning capabilities and is more compute-efficient than its predecessor PaLM. |
| LIMA: Less Is More for Alignment |
Chunting Zhou et. al. |
2023-05-18 |
Large language models are trained in two stages: (1) unsupervised pretraining from raw text, to learn general-purpose representations, and (2) large scale instruction tuning and reinforcement learning, to better align to end tasks and user preferences. We measure the relative importance of these two stages. |
| Training Compute-Optimal Large Language Models |
Jordan Hoffmann et. al. |
2022-03-29 |
We investigate the optimal model size and number of tokens for training a transformer language model under a given compute budget. |