Select runs that logged avg_loss
to visualize data in this line chart.
to visualize data in this line chart.
universal decay
Created on August 21|Last edited on August 22
Comment
Allow the plasticity to reach as low as -plast_clip. in this case, that's 1e2.
Back to the lower cap for plasticity being 1e-15. Really just test if the decay value can be higher than plasticity.
have the decay match the lr.
compare
I went on to define a larger gap between the lr and the decay.
try:
- very small layer size - vsmalllayer
- much larger layer size - vlargelayer
- larger decay rate - largedecay
- incorporate negative plasticity again. - negplast
- tinker with a significantly smaller plasticity
Add a comment