I've long suspected that there was smth wrong w backprop

I've long suspected that there was smth wrong w backprop

some features may be unnecessary.

An attempt to make testing palindromes fast. Just breakneck lr, pretty much.

just a wee bit o hyperparameter tuning. nothing serious

Starts with the Key-Recall task. Then, perhaps the associative retrieval task from Ba & Hinton, repeated sequences, or even the baby names or short stories datasets. Might consider hyperparameter sweeps or a search for the ideal effective update magnitude.

a new mechanism keeps the average of the plasticity values to be around 1, so that the effective lr is always the same, overall.

I want to just tinker with what I have so far to just get the model to begin to really show useful results on this dataset. The build of it really feels like it inherently proves what I'm seeking to prove about the model.

experiments with static plasticity and with trainable

Should the plasticity values for each parameter change? Or can they stay the same and still let the model work fine? What should their value be? Randomly initialized, or all zero, or something else?

Remove the recurrent connection. Try to get the rnn running, anyways, using weight updates for short-term memory. The plasticity value ought to be instrumental, here, some weights being more plastic than others and therefore changing quickly so as to store info.

a look at what the rnn is doing step-by-step

Direct feedback alignment, but the weight updated is modulated by a local rule that preserves signal from the previous layer.