2

I've been trying to perform the following differentiation of a neural network:

$$\frac{\delta||h(XW)\alpha-y||^2}{\delta W} = \frac{\delta}{\delta W}\sum_i(h(XW)_i\alpha-y_i)^2$$

Where $X$ and $W$ and matrices, $\alpha$ and $y$ are vectors, and $h$ is a point wise applied function.

I've been reading the Wikipedia article on Matrix calculus and "The Matrix Cookbook" all day, but I can't seem to get things to work. I think it should probably be

$$2(h(XW)\alpha-y)\frac{\delta}{\delta W}(h(XW)\alpha)$$

But I certainly get stuck at the $h$ function, which I guess you could say is from matrix to matrix.

Any hints would be appreciated.

Update: I think this derivation is correct:

$$\frac{\delta||h(XW)\alpha-y||^2}{\delta W} = \frac{\delta}{\delta W}(h(XW)\alpha-y)^T(h(XW)\alpha-y) = 2(h(XW)\alpha-y)^T\frac{\delta}{\delta W}(h(XW)\alpha) = 2(h(XW)\alpha-y)^TX^Th'(XW)\alpha$$

This was derived by derivating over each element of $W$ using traces.

Update 2: I found this great presentation of the topic: Schonemann_Trace_Derivatives_Presentation.pdf which I recommend very much.

I've reformulated by defining $H=h(XW)$, $H'=h'(XW)$, $E = HA-Y$. Hence the problem has the pretty solution

$$\frac{\delta}{\delta W}||E||_F^2 = \frac{\delta}{\delta W}Tr(EE^T) = 2X^T(H' \odot EA^T)$$

Where $\odot$ is the point wise product. By working more with the trace manipulations you can also get a formula for the generalized problem $h(h(...)W_2)W_1)W_0$.

Thomas Ahle
  • 5,925
  • Did you mean "...$X$ and $W$ are matrices..." ? – pitchounet Mar 12 '14 at 09:35
  • You are right. Thanks, I've corrected it. – Thomas Ahle Mar 12 '14 at 09:50
  • Thomas you are not serious. Your update 2 has nothing to do with your question. OK, I stop wasting my time. –  Mar 15 '14 at 10:06
  • loup, I don't understand. All I did was moving from vectors to matrices, so I could formulate the norm using traces. It's a pure generalization when considering vectors as one dimensional matrices. – Thomas Ahle Mar 15 '14 at 13:54
  • Ah yes, it is a generalization ! Now, since we have a trace, using my answer, I can calculate the gradient ; one finds $\nabla_W(f)=2X^TH'^TEA^T$. –  Mar 15 '14 at 14:58

3 Answers3

2

In its second updated form, the function and its differential can be written in terms of the Frobenius product as $$ \eqalign { f &= E:E \cr df &= 2\,E:dE \cr &= 2\,E:dHA \cr &= 2\,EA^T:dH \cr &= 2\,EA^T:H'\circ d(XW) \cr &= 2\,EA^T\circ H':XdW \cr &= 2\,X^T(EA^T\circ H'):dW \cr } $$ Yielding the derivative as $$ \eqalign { \frac{\partial f}{\partial W} &= 2\,X^T(EA^T\circ H') \cr } $$ The Hadamard product is commutative, so this equals Thomas' result.

greg
  • 196
  • 5
  • The ${E,A}$ variables are specified to be vectors, therefore the Hadamard product can be eliminated in favor of diagonal matrices, yielding $$ \frac{\partial f}{\partial W} = 2X^T\cdot{\rm Diag}(E)\cdot H'\cdot{\rm Diag}(A) $$ – greg Feb 14 '22 at 14:27
1

Thomas, here there are no traces. I assume that $h:M_n\rightarrow M_n$ and $||.||$ is the euclidean norm on $\mathbb{R}^n$. The derivative of $h$, in the point $XW$, is denoted by $D_{XW}h\in L(M_n,M_n)$ and is a jacobian. If $f:W\rightarrow ||h(XW)\alpha-y||^2$, then $D_{W}f:H\in M_n\rightarrow \mathbb{R}$ is a linear application. If $g:W\rightarrow h(XW)\alpha-y$, then $D_Wg:H\rightarrow D_{XW}h(XH)\alpha$. Finally the required derivative $D_Wf:H\rightarrow 2(h(XW)\alpha-y)^T(D_{XW}h)XH\alpha$ doesn't have a nice head.

$f$ is a real function and then, has a gradient that (theoretically) we can write using the matrix scalar product $(U,V)=trace(U^TV)$. Unfortunately, "trace" does not appear in the formula giving $D_Wf$. I'm afraid that you have to settle for the previous result.

0

Here's a definition using tensor diagrams. We first define $E$ as before: E def Now $\|E\|_2^2$ is just $E$ multiplied with $E$ along both axes. So we can take the derivative using the product rule: d/dW E Tensor diagrams make it easy to handle $dE/dW$ in isolation, since tensors of arbitrary orders can be easily represented. Here we use the chain rule for point-wise applied functions: dE Putting it all together and translating back to standard notation (optional), we get: final I tested this numerically, and it matches what we get using matrixcalculus.org: mc So I think having $X^T$ is actually correct, and the previous result was wrong.

Thomas Ahle
  • 5,925