LaTeX Fiddles

Created on February 11|Last edited on February 11

Comment

﻿
UnderbracesThe layerwise gradients of a deep linear neural network have the following form:
∇Wkl⏟grad w.r.t. layer k(W1,…,WD)=Wk+1:⊤⏟later weights, adjoint   ∇L(W)⏟collapsed loss   W:k⏟earlier weights\underbrace{\nabla{_{W_k}}l}_\text{grad w.r.t. layer \textit{k}}\left(W_1, \dots, W_D\right)

= \underbrace{W_{k+1:}^\top}_
\text{later weights, adjoint} \ \ \ 
\underbrace{\nabla \mathcal{L}\left(W\right)}
_\text{collapsed loss} \ \ \ 
\underbrace{W_{:k}}_\text{earlier weights}grad w.r.t. layer k∇Wk​​l​​(W1​,…,WD​)=later weights, adjointWk+1:⊤​​​   collapsed loss∇L(W)​​   earlier weightsW:k​​​﻿
﻿

Add a comment