MoRA: High-Rank Updating for Parameter-Efficient Fine-Tuning

🌈 Abstract

The paper introduces a new method called MoRA (High-Rank Updating for Parameter-Efficient Fine-Tuning) that employs a square matrix instead of low-rank matrices to achieve high-rank updating while maintaining the same number of trainable parameters as LoRA. The authors analyze the limitations of low-rank updating in LoRA, particularly for memory-intensive tasks, and propose several non-parameter operators to reduce the input dimension and increase the output dimension for the square matrix. The authors evaluate MoRA across five tasks - memory, instruction tuning, mathematical reasoning, continual pretraining, and pretraining - and find that MoRA outperforms LoRA on memory-intensive tasks while achieving comparable performance on other tasks.

🙋 Q&A

[01] Introduction

1. What is the key idea behind MoRA? The key idea of MoRA is to use a square matrix instead of low-rank matrices to achieve high-rank updating while maintaining the same number of trainable parameters as LoRA.

2. What are the limitations of low-rank updating observed in LoRA? The authors observe that low-rank updating in LoRA struggles with tasks that require enhancing knowledge and capabilities through fine-tuning, such as complex reasoning or continual pretraining, compared to tasks that primarily focus on interacting with the format, like instruction tuning.

3. How does MoRA aim to address the limitations of low-rank updating? MoRA employs a square matrix instead of low-rank matrices to maximize the rank while maintaining the same number of trainable parameters. The authors also develop corresponding non-parameter operators to decrease the input dimension and increase the output dimension for the square matrix.

[02] Analysis of the Influence of Low-rank Updating

1. What experiment did the authors conduct to demonstrate the limitations of low-rank updating? The authors conducted an experiment on memorizing Universally Unique Identifiers (UUIDs) to assess the performance of LoRA and FFT in memorizing new knowledge, bypassing the use of the original knowledge of the language model.

2. What were the key findings from the memory task experiment? The authors found that low-rank updating in LoRA struggled to memorize new knowledge compared to FFT, even when increasing the rank of LoRA. In contrast, LoRA matched the performance of FFT in instruction tuning, which primarily leverages the original knowledge of the language model.

[03] MoRA Method

1. What is the key idea behind the square matrix used in MoRA? The key idea is to use a square matrix instead of two low-rank matrices in LoRA to achieve a higher rank in the weight update while maintaining the same number of trainable parameters.

2. How do the non-parameter operators in MoRA work to reduce the input dimension and increase the output dimension for the square matrix? The authors explore several methods to implement the non-parameter operators, including truncation, sharing rows and columns, and reshaping the square matrix. These operators aim to reduce the input dimension and increase the output dimension for the square matrix while ensuring that the weight can be merged back into the language model like LoRA.

3. How does the rotation operator in MoRA help improve the expressiveness of the square matrix? The authors incorporate a rotation operator inspired by RoPE to enable the square matrix to differentiate between the various inputs, further augmenting the expressiveness of the square matrix.

[04] Experiments

1. What are the key fine-tuning tasks used to evaluate MoRA, LoRA, and other baselines? The authors evaluate the methods across three fine-tuning tasks: instruction tuning, mathematical reasoning, and continual pretraining.

2. What are the key findings from the fine-tuning task results? The authors find that MoRA matches the performance of LoRA on instruction tuning and mathematical reasoning, but outperforms LoRA on continual pretraining and memory-intensive tasks. The authors also observe that LoRA variants exhibit similar performances across the evaluated tasks.

3. How do the pretraining results further demonstrate the effectiveness of high-rank updating in MoRA? The pretraining experiments show that MoRA and its variant ReMoRA, which merges the square matrix back into the original parameters during training, outperform LoRA and ReLoRA in terms of pretraining loss and perplexity on the C4 dataset.

</output_format>