Select runs that logged avg_loss
to visualize data in this line chart.
to visualize data in this line chart.
reg the plast again
Created on September 3|Last edited on September 5
Comment
l1 regularization, now based off what I read in this paper: https://www.semanticscholar.org/paper/Oja%27s-plasticity-rule-overcomes-several-challenges-Shervani-Tabar-Mirhoseini/abff9e22c9eacc88d33d3b1fe5eef22af7a9fbfb?utm_source=alert_email&utm_content=LibraryFolder&utm_campaign=AlertEmails_WEEKLY&utm_term=LibraryFolder&email_index=0-0-0&utm_medium=39383040
lr, plr, and plast_clip are swept, here.
now, the code is modified to use `torch.exp(plasticity) instead of just plasticity.
now for the smaller models, for speed and visibility
Add a comment