Is the gradient really the transpose of the Jacobian for scalar-valued function, or is there something deeper?

Question

I first learned about the gradient through an optimization text, and in that text the gradient vector at $x$ for a scalar-valued differentiable function $f: D \to \mathbb{R}$ is always treated as a column vector of the partials. The transpose of $\nabla f(x)$ is the Jacobian of $f$ at $x$.

But ever since I've dealt with dynamical systems, I found that people likes to treat $\nabla f(x)$ as a row vector and in which case it is exactly the same as the Jacobian.

Is this simply a convention between fields? Or is there something deeper going on?

I know that both the gradient and the Jacobian comes from the derivative (or the total derivative, or the Frechet derivative). They are basically children of this concept. Since there is a single concept and the total derivative BECOMES either the gradient or the Jacobian depending on context, therefore both the structure of the gradient and the Jacobian should be the same, no? So what's up with the transpose?

Answer 1

To keep things simple let's consider a differentiable function $f : V \to \mathbb{R}$ where $V$ is an abstract finite-dimensional real vector space. The difference between the Jacobian / differential and the gradient is that the Jacobian can be defined using no further structure and is a covector (element of the dual $V^{\ast}$), while defining the gradient requires an inner product on $V$ and is a vector (element of $V$). This is why the gradient can be used to move around in $V$ in gradient descent, for example; the Jacobian cannot be used in this way.

Formally, the Jacobian at a point $p \in V$ is a linear map $df_p : T_p(V) \to \mathbb{R}$ from the tangent space $T_p(V)$ at $p$ to $\mathbb{R}$. Since $V$ is a vector space all of its tangent spaces can canonically be identified with $V$, so $df_p$ is a linear map $V \to \mathbb{R}$, which is exactly a covector in $V^{\ast}$. So it transforms in a different way than elements of $V$ transform under arbitrary change of coordinates; for example, if elements of $V$ are scaled up, elements of $V^{\ast}$ are scaled down.

If $V$ is equipped with an inner product $\langle -, - \rangle$ (for example the standard Euclidean dot product on $\mathbb{R}^n$) then this identifies vectors $v$ and covectors $\langle v, - \rangle$, giving an isomorphism $V^{\ast} \cong V$ (the musical isomorphism) which we use to convert the covector $df_p$ into a vector; this is the gradient $\nabla f_p$, so by definition it satisfies

$$\langle \nabla f_p, w \rangle = df_p(w)$$

where $w \in T_p(V) \cong V$ is a tangent vector. This is what taking the transpose "really means"; note that this conversion now depends on the choice of inner product and changes (in the sense that you will get different vectors) if you choose a different inner product.

More globally and abstractly we can talk about differentiable functions $f : M \to \mathbb{R}$ where $M$ is a differentiable manifold. Then the Jacobian at a point $p \in M$ is a linear map $df_p : T_p(M) \to \mathbb{R}$, hence an element of the cotangent space $T_p(M)^{\ast}$, and can be defined using no additional structure. Globally the entire differential $df$ assembles into a differential form. Defining the gradient requires a Riemannian metric converting these cotangent vectors into tangent vectors $\nabla f_p \in T_p(M)$, which assemble into a vector field $\nabla f$. This allows us to discuss gradient descent, gradient flow, etc.

Answer 2

Here's a simpler answer for those not familiar with advanced differential geometry.

In brief the Jacobian is a matrix, the gradient is a vector.

Recall the following definition.

$f : U \subset \mathbb R^n \to \mathbb R^m $ is differentiable in $p \in U$ if and only if there exists a linear transform $L_p : \mathbb R^n \to \mathbb R^m$ such that $$ \lim_{x \to p} \frac{\Vert f(x)-f(p)- L(x-p)\Vert}{\Vert x-p\Vert} = 0.$$ This linear transform is often denoted as $df_p := L_p$ and it is the best linear approximation of $f$ around $p$. In particular if $f$ is already linear then it is its own best linear approximation so $df_p = f$.

---

This linear transformation can be viewed as a matrix which is the jacobian and when $m=1$ we have a matrix multiplied with a vector

$$ df_p (x) = \begin{bmatrix} \frac{\partial f(p)}{\partial x_1} & \dots & \frac{\partial f(p)}{\partial x_n} \end{bmatrix} \begin{bmatrix} x_1 \\ \vdots \\ x_n \end{bmatrix}$$

So the jacobian is a "row vector" (it's not really a vector, it is a matrix) Moreover the last expression is exactly the dot product of two coordinate vectors (no matrices here).

$$ df_p (x) = \begin{bmatrix} \frac{\partial f(p)}{\partial x_1} \\ \vdots \\ \frac{\partial f(p)}{\partial x_n} \end{bmatrix} \cdot \begin{bmatrix} x_1 \\ \vdots \\ x_n \end{bmatrix} $$

Here the gradient is a vector whose coordinates are the partial derivatives

Duality

From a linear algebra POV we know that when $m = 1$ then the linear map $df_p $ is just a linear map from $V = \mathbb R^n \to \mathbb R$. That is, $df_p\in V^*$ is an element of the dual space.

So when fixing a basis $e_1,...,e_n$ of $V$ this also defines a dual basis $e_1^*, e_2^*,..., e_n^*$ on the vector space $V^*$.

Note that these are exactly the coordinate maps $$ e_i^* : \mathbb R^n \to \mathbb R : (x_1,...,x_n) \mapsto x_i.$$

That is, $e_i^*$ is just a map which extracts the i-th coordinate of a coordinate vector $x =(x_1,...,x_n)$. That is, $e_i^*(x) = x_i.$

However we know that $$ df_p(x) = \sum_{i = 1}^n \frac{\partial f(p)}{\partial x_i} x_i $$

so $$ df_p = \sum_{i = 1}^n \frac{\partial f(p)}{\partial x_i} e_i^*.$$

In particular using the isomorphism between $V$ and $V^*$ we have that $df_p$ is exactly the dual vector of the vector in $V$ with coordinates

$$ \left(\frac{\partial f(p)}{\partial x_1},..., \frac{\partial f(p)}{\partial x_n} \right)$$

Again we see that the gradient is a vector in $V$. And yes something deeper is going on:

• the gradient and the differential are dual as elements in $V$ and $V^*$ respectively.

As a bonus we also see that since $e_i^* = x^i$ is a linear map it is its own best linear approximation so $d(x^i) = x^i$ and we have the formula

$$ df_p = \sum_{i = 1}^n \frac{\partial f}{\partial x_i} dx_p^i$$

transpose

By equipping $V$ with a dot product we have $$ (df_p)(x) = \nabla f \cdot_V x.$$ The transpose is just a map $(df_p)^t : \mathbb R \to V$ and we use the dot product to identify the value of $(df_p)^t(\alpha)$ as follows.

$$df_p(x) \cdot_{\mathbb{R}} \alpha = \nabla f \cdot_V x \alpha = x \cdot_V \underbrace{(\nabla f \alpha)}_{df_p^t(\alpha)}.$$

Since this is a linear map it has a matrix. It follows that the matrix of the transpose is just acting on a scalar by multiplying it by the an $n \times 1$ matrix.

$$ (df_p)^t (\alpha) = \begin{bmatrix} \frac{\partial f(p)}{\partial x_1} \\ \vdots \\ \frac{\partial f(p)}{\partial x_n} \end{bmatrix}\begin{bmatrix} \alpha \end{bmatrix} $$

So yes, the transpose of the jacobian matrix is a column matrix whose coefficients are those of the gradient.