Pythia 1B Trained Parameters LoRA Fine-tuning Compare
Created on May 19|Last edited on May 19
Comment
Compare Trained Parameters
Comparing LoRA fine-tuning with ShareGPT dataset, mixed English and Chinese.
- Brown: Fine-tune after training only embed (embed_in.weight, embed_out.weight).
- Green: Fine-tune after training embed + attention (all above + layers.n.post_attention_layernorm.weight, layers.n.post_attention_layernorm.bias, layers.n.attention.query_key_value.weight, layers.n.attention.query_key_value.bias, layers.n.attention.dense.weight, layers.n.attention.dense.bias).
- Dark Red: Fine-tune after training all parameters.
Run set
3
Add a comment