Eigen Decomposition

Eigen Decomposition is extremely useful for square symmetric matrices.

The physical interpretation can be given as: "Every real matrix can be thought of as a combination of rotation and stretching."

$$A\overrightarrow{v} = \overrightarrow{w} \\ A \in \mathbb{R}^{n \times n} \hspace{0.5cm} \overrightarrow{v} \in \mathbb{R}^{n \times 1} \hspace{0.5cm} \overrightarrow{w} \in \mathbb{R}^{n \times 1}$$

Eigen Vectors

Eigen Vectors of a matrix are those special vectors that only stretch under the action of a matrix.

Eigen Values

Eigen Values are the factor by which eigen vectors stretch.

$$Av = \lambda v$$

Eigen Decomposition Derivation​

Eigen Decomposition is that set of vectors which only stretch under the action of matrix $A$​. Let's say, $A (\mathbb{R}^{n \times n})$​ has $n$ linearly independent eigenvectors:

$$\begin{Bmatrix} v^1, v^2, v^3, ..., v^n\\ \end{Bmatrix}$$

​Concatenate all the vectors (as columns) and make a eigen vector matrix $V$​.

$$V = \begin{bmatrix} v^1 & v^2 & ... & v^n\\ \end{bmatrix}$$

Then, we concatenate the corresponding eigen values into a diagonal matrix:

$$\Lambda = diag(\lambda_1, \lambda_2, ..., \lambda_n)$$

Eigen Decomposition ​(Factorisation) of $A$​ is given by the following equation:

$$A = V\Lambda V^{-1}$$

For real symmetric matrices, we have real eigen vectors and values:

$$A = Q \Lambda Q^T \hspace{0.3cm} where \hspace{0.1cm} Q^T = Q^{-1}$$

Thus, $Q$ is orthogonal. This also means $Q$​ is a rotation matrix. So, for the following action:

$$Av = Q \Lambda Q^Tv$$

​$Q^Tv$​ rotates the vector $v$​ in the direction of the eigen vector. Then, $\Lambda(Q^Tv)$​ stretches the vector $v$​. Finally, $Q (\Lambda Q^Tv)$​ rotates $v$​ back to its original direction.

Note: Eigen Decomposition may not be unique.