If we think of a matrix as a description of some linear map with respect to some bases, then the transpose matrix is a description of the dual map with respect to the dual bases. A permutation $\sigma$ of a basis induces the inverse permutation $\sigma^{-1}$ on the dual basis. This gives a conceptual explanation of the fact that the transpose of a permutation matrix is its inverse.
For a vector space $V$ over a field $k$, let $V^\ast$ denote the dual space — the space of linear maps from $V$ to $k$.
For a linear map $L:V\rightarrow W$, there is a dual map $L^\ast:W^\ast\rightarrow V^\ast$ defined in the following way:
$$L^\ast(f)=f\circ L$$
Let $V$ be a finite-dimensional vector space with basis $\mathcal{B}$. For each $v_i\in\mathcal{B}$, define $v_i^\ast\in V^\ast$ to be the linear functional such that:
$$v_i^\ast(v_j)=\delta_{ij}$$
Let $\mathcal{B}^\ast=\{v_i^\ast\}_{1\leq i\leq n}$. Then $\mathcal{B}^\ast$ forms a basis for the dual space $V^\ast$.
Let $L:V\rightarrow W$ be a linear map. Let $\mathcal{A}$ and $\mathcal{B}$ be bases for $V$ and $W$ respectively, and let $L_{\mathcal{AB}}$ be the matrix of $L$ with respect to $\mathcal{A}$ and $\mathcal{B}$. You may check that:
$$L_{\mathcal{B^\ast A^\ast}}=L_{\mathcal{AB}}^T$$
Once again, let $\mathcal{B}=\{v_i\}_{1\leq i\leq n}$ be a basis for $V$, and let $\mathcal{B}^\ast$ be the dual basis. Let $\sigma:\{1,2,\dots, n\}\rightarrow\{1,2,\dots, n\}$ be a permutation, and let $P_\sigma:V\rightarrow V$ be the linear map such that $P_\sigma(v_i)=v_{\sigma(i)}$. The matrix of $P_\sigma$ with respect to $\mathcal{B}$ is the permutation matrix representing $\sigma$.
It is easily checked that:
$$P_{\sigma}^\ast(v_i^\ast)=v_{\sigma^{-1}(i)}^\ast$$
Thus, the transpose of the permutation matrix representing $\sigma$ is its inverse.