reg the plast again

Created on September 3|Last edited on September 5
Comment
l1 regularization, now based off what I read in this paper: https://www.semanticscholar.org/paper/Oja%27s-plasticity-rule-overcomes-several-challenges-Shervani-Tabar-Mirhoseini/abff9e22c9eacc88d33d3b1fe5eef22af7a9fbfb?utm_source=alert_email&utm_content=LibraryFolder&utm_campaign=AlertEmails_WEEKLY&utm_term=LibraryFolder&email_index=0-0-0&utm_medium=39383040﻿
lr, plr, and plast_clip are swept, here.
﻿
avg_loss
avg_loss
02k4k6k8kStep0.811.21.41.61.82
0.01   10   100
0.01   10   10
0.01   0.001   100
0.001   100   100
0.001   10   100
0.001   0.1   100
0.0001   1   100000
0.0001   0.1   10000
0.0001   0.1   10000
0.0001   0.1   100
0.0001   0.01   100000
0.0001   0.01   100000
0.0001   0.01   100000
0.0001   0.01   10000
0.0001   0.01   10000
0.0001   0.01   10000
0.0001   0.01   10000
0.0001   0.001   1000000
0.0001   0.001   1000000
0.0001   0.001   10000
0.00001   0.01   100000
Run set22
﻿
now, the code is modified to use `torch.exp(plasticity) instead of just plasticity.
﻿
Run set28
﻿
now for the smaller models, for speed and visibility
﻿
Run set18
﻿
﻿
﻿
Run set4
﻿
﻿
Add a comment