Select runs that logged avg_loss
to visualize data in this line chart.
to visualize data in this line chart.
back to basics - static
Created on September 5|Last edited on September 11
Comment
plasticity is uniform dist, 0 to 2.
other lr
try 0 to 10.
try exp -2 to 4
set a minimum plast.
Static plasticity. Randomly initialized. DFA weights are normally distributed now. Sigmoid activation to avoid exploding gradients (the main problem I've been dealing with for months).
I suspect that the majority of stability is provided by the new activation. It's rather a drastic choice.
test the limits of how high and how wide the range plasticity can be.
what about a normal distribution of values?
chill out. no need for exp() on the plasticity values. the distribution can be less extreme.
Add a comment