0

You often see vector-matrix algebra derivations in which some vector or matrix variables introduced are transposed along the way. It seems necessary to match the inner dimensions for calculation, but why is it permissible? Is there an intuitive way to explain this?

Sorry for the lack of certainty. A specific example. Consider gradient descent method orthogonality of search steps. $$X^{n+1}=X^{n}+\alpha \nabla f(X^{n})$$ $$\frac{\partial f(X^{n+1})}{\partial \alpha} = 0$$ $$\frac{\partial f(X^{n}+\alpha \nabla f(X^{n}))}{\partial (X^{n}+\alpha \nabla f(X^{n})} . \frac{\partial (X^{n}+\alpha \nabla f(X^{n}))}{\partial \alpha} = 0$$

Leads to $$(\nabla f(X^{n+1}))^{T} . \nabla f(X^{n}) = 0$$

Why is $(\nabla f(X^{n+1}))$ transposed?

user1011182
  • 175
  • 8
    Without content, it's hard to tell what question you have. Moreover, any column vector $A\in \mathbb{F}^{n\times 1}$ will not equal to its transpose $A^\top \in \mathbb{F}^{1\times n}$ as long as $n > 1$. Can't really see what point you're trying to make for this "counter-example." – pbb Apr 19 '24 at 14:46
  • 3
    Can you give an example of an actual calculation someone has done in which you're not sure if the introduction of a transpose is permissible? Sure $A \ne A^{\mathsf T}$, but on the other hand it is permissible to multiply both sides of an equation by $A$ or both sides by $A^{\mathsf T}$, and there we are really free to do whatever makes the dimensions work out. – Misha Lavrov Apr 19 '24 at 14:48
  • Thank u for the comments. how about if we consider optimum gradient method. jumping in to some math after a taylor series expansion we have,

    $$f(Y^{k}+\beta S^{k}) = f(Y^{k})+\nabla f(Y^{k})\beta S^{k} + 0.5(\beta S^{k})^{T} A\beta S^{k}$$ $$\frac{df(Y^{k}+\beta S^{k})}{d\beta}=0$$ $$\nabla f(Y^{k})^{T}S^{k}+(S^{k})^{T}AS^{k}\beta=0$$

     – user1011182 Apr 19 '24 at 15:15
  • thanks misha. yes, i think this is what i have missed. – user1011182 Apr 19 '24 at 15:17
  • 3
    @user1011182 okay, at least we're starting somewhere. What about you write down this example more formally in your main question rather than just throwing three equations without any definition of the variables? I guess that will make your question a bit clearer. – pbb Apr 19 '24 at 15:35

2 Answers2

1

Here's an example from a course I took. Let

$$ \Phi(y)=\frac{1}{2} y^{\top} A y-y^{\top} b. $$

Let's take the gradient. By linearity,

$$\nabla \Phi(y)=\frac{1}{2}\nabla y^TAy - \nabla y^Tb$$

Now by the product rule for the gradient, we have

$$=\frac{1}{2}\left[Ay + y^TA\right]- \nabla y^Tb$$

but since $y^TA = A^Ty$ (Since $(AB)^T=B^TA^T$ and $(A^T)^T=A)$, we can write this as

$$\frac{1}{2}(A+A^T)y - \nabla y^Tb$$

and now, $\nabla y^Tb = b$ so in conclusion, we have

$$ \nabla \Phi(y)=\frac{1}{2}\left(A^{\top}+A\right) y-b. $$

Thus, we see that $A^T$ appears naturally out of the gradient calculation. So we aren't just choosing to use $A^T$, it has to be there.

Then as it turns out, in this particular problem, $A$ was symmetric, so in your words, we "freely" transpose it to obtain $Ay-b$. However, the case where $A$ is non-symmetric is a good illustration of a transpose appearing and having to be there.

If you're confused about what the transpose is abstractly, I suggest you learn about dual spaces and adjoints.

The only time we can "freely" transpose a matrix is when the matrix is symmetric, that is $A^T=A$.

random0620
  • 3,375
1

Based on your comment (which is not so clear but suffices a guess), I believe your concern might be you don't know why sometimes you need to transpose vectors in the gradient calculation. Going through some matrix calculus should help. Nothing really deep here: it's just a convention of either using the numerator layout or the denominator layout, and the calculation works out that way.

pbb
  • 395
  • 1
    This was also my read on the question. I wondered this myself when I was trying to watch a lecture on support vector machines in highschool before I studied math in college. The lecturer transposed things like it was nothing! But now I see that he had already done the calculation ahead of time and you cant just transpose things "freely" as OP puts it. – random0620 Apr 19 '24 at 15:56
  • 1
    https://www.youtube.com/watch?v=_PwhiWxHK8o Heres the lecture. At some point he forgot to transpose something and he wildly slaps some transposes in there. It makes it look like its a "choice" but its not. The transpose has to be there – random0620 Apr 19 '24 at 15:59