Let $\mathbf{A} \in \mathbb{R}^{m \times n}$ and $\mathbf{B} \in \mathbb{R}^{p \times q}$. The $mp \times nq$ matrix defined by $$ \mathbf{A} \otimes \mathbf{B} = \begin{pmatrix} a_{11} \mathbf{B} & \cdots & a_{1n} \mathbf{B} \\ \vdots & & \vdots \\ a_{m1} \mathbf{B} & \cdots & a_{mn} \mathbf{B} \end{pmatrix} $$ is called the Kronecker product of $\mathbf{A}$ and $\mathbf{B}$.
Note Kronecker product is defined for any pair of matices of general shape.
Following facts are easy to verify.
If $\mathbf{A}$ and $\mathbf{B}$ are non-singular, then $$ (\mathbf{A} \otimes \mathbf{B})^{-1} = \mathbf{A}^{-1} \otimes \mathbf{B}^{-1}. $$ Same identity holds for generalized inverses as well.
These formulas are useful for reducing computations on Kronecker product to those on original (much smaller) matrices.
Eigenvalues of Kronecker product. Let $\mathbf{A}$ be an $m \times m$ matrix with eigenvalues $\lambda_1,\ldots, \lambda_m$, and $\mathbf{B}$ a $p \times p$ matrix with eigenvalues $\mu_1, \ldots, \mu_p$. Then the $mp$ eigenvalues of $\mathbf{A} \otimes \mathbf{B}$ are $\lambda_i \mu_j$, $i=1,\ldots,m$, $j=1,\ldots,p$.
using LinearAlgebra, Random
Random.seed!(216)
m, p = 3, 5
# two symmetric matrices
A = randn(m, m); A = (A + A') / 2
B = randn(p, p); B = (B + B') / 2
AkronB = kron(A, B)
Aeig = eigen(A)
Beig = eigen(B)
eigen(AkronB)
[Aeig.values[i] * Beig.values[j] for i in 1:3, j in 1:5]
If $\mathbf{x}_1$ is an eigenvector of $\mathbf{A}$ and $\mathbf{x}_2$ an eigenvector of $\mathbf{B}$, then $\mathbf{x}_1 \otimes \mathbf{x}_2$ is an eigenvector of $\mathbf{A} \otimes \mathbf{B}$. It is not generally true that every eigenvector of $\mathbf{A} \otimes \mathbf{B}$ is the Kronecker product of an eigenvector of $\mathbf{A}$ and an eigenvector of $\mathbf{B}$.
Example:
$$
\mathbf{A} = \mathbf{B} = \begin{pmatrix} 0 & 1 \\ 0 & 0 \end{pmatrix}, \quad \mathbf{e}_1 = \begin{pmatrix} 1 \\ 0 \end{pmatrix}, \quad \mathbf{e}_2 = \begin{pmatrix} 0 \\ 1 \end{pmatrix}.
$$
$\mathbf{e}_1$ is the only eigenvector of $\mathbf{A}$ and $\mathbf{B}$. But the eigenvector of $\mathbf{A} \otimes \mathbf{B}$ are not just $\mathbf{e}_1 \otimes \mathbf{e}_1$, but also $\mathbf{e}_1 \otimes \mathbf{e}_2$ and $\mathbf{e}_2 \times \mathbf{e}_1$.
Corollaries:
Definiteness of Kronecker product. If $\mathbf{A}$ and $\mathbf{B}$ are positive (semi)definite, then $\mathbf{A} \otimes \mathbf{B}$ is positive (semi)definite.
Determinnant of Kronecker product. For $\mathbf{A} \in \mathbb{R}^{m \times m}$ and $\mathbf{B} \in \mathbb{R}^{p \times p}$, $$ \text{det}(\mathbf{A} \otimes \mathbf{B}) = \text{det}(\mathbf{A})^p \text{det}(\mathbf{B})^m. $$
Rank of Kronecker product. $$ \text{rank}(\mathbf{A} \otimes \mathbf{B}) = \text{rank}(\mathbf{A}) \text{rank}(\mathbf{B}). $$
Let $\mathbf{A} \in \mathbb{R}^{m \times m}$ and $\mathbf{B} \in \mathbb{R}^{n \times n}$. The Kronecker sum is defined as the $mn \times mn$ matrix: $$ \mathbf{A} \oplus \mathbf{B} = \mathbf{A} \otimes \mathbf{I}_n + \mathbf{I}_m \otimes \mathbf{B}. $$
The eigenvalues of Kronecker sum $\mathbf{A} \oplus \mathbf{B}$ are $\lambda_i + \mu_j$, where $\lambda_i$ and $\mu_j$ are eigenvalues of $\mathbf{A}$ and $\mathbf{B}$ respectively.
The vec operator stacks the columns of a matrix into one long vector. For $$ \mathbf{A} = \begin{pmatrix} \mid & & \mid \\ \mathbf{a}_1 & \cdots & \mathbf{a}_n \\ \mid & & \mid \end{pmatrix} \in \mathbb{R}^{m \times n}, $$ we have $$ \text{vec} \, \mathbf{A} = \begin{pmatrix} \mathbf{a}_1 \\ \vdots \\ \mathbf{a}_n \end{pmatrix}. $$
Following facts are easy to verify:
$$
\text{vec} \, \mathbf{a} \mathbf{b}' = \mathbf{b} \otimes \mathbf{a}.
$$
$$
(\text{vec} \, \mathbf{A})' \text{vec} \, \mathbf{B} = \text{tr}(\mathbf{A}' \mathbf{B}).
$$
Let $\mathbf{A}$, $\mathbf{B}$ and $\mathbf{C}$ such that the matrix product $\mathbf{A} \mathbf{B} \mathbf{C}$ is defined. Then $$ \text{vec} \, \mathbf{A} \mathbf{B} \mathbf{C} = (\mathbf{C}' \otimes \mathbf{A}) \text{vec} \, \mathbf{B}. $$
Because of the extreme usefulness of this identity, we give it a name: Roth’s Kronecker product identity, or Roth column lemma, or simply the vec trick.
A generalization to the product of 4 matrices: $$ \text{tr} \, \mathbf{A} \mathbf{B} \mathbf{C} \mathbf{D} = (\text{vec} \, \mathbf{D}' )' (\mathbf{C}' \otimes \mathbf{A}) (\text{vec} \, \mathbf{B}) = (\text{vec} \, \mathbf{D})' (\mathbf{A} \otimes \mathbf{C}') (\text{vec} \, \mathbf{B}'). $$
using LinearAlgebra, Permutations
vech(A) = [A[i, j] for i in 1:size(A, 1), j in 1:size(A, 2) if i ≥ j]
A = randn(3, 3)
vech(A)
Let $\mathbf{A} \in \mathbb{R}^{m \times n}$. $\text{vec} \, \mathbf{A}$ and $\text{vec} \, \mathbf{A}'$ clearly contain the same $mn$ components. There exists a unique $mn \times mn$ permutation matrix that transforms $\text{vec} \, \mathbf{A}$ to $\text{vec} \, \mathbf{A}'$. We denote it $\mathbf{K}_{mn}$ $$ \mathbf{K}_{mn} \text{vec} \, \mathbf{A} = \text{vec} \mathbf{A}'. $$
Since $\mathbf{K}_{mn}$ is an $mn \times mn$ permutation matrix, it is orthogonal, i.e., $$ \mathbf{K}_{mn}^{-1} = \mathbf{K}_{mn}' = \mathbf{K}_{nm}. $$ Also $$ \mathbf{K}_{n1} = \mathbf{K}_{1n} = \mathbf{I}_n. $$
A = [1 3 5; 2 4 6]
vec(A)
vec(A')
K23 = Matrix(inv(Permutation(vec(A'))))
K23 * vec(A) == vec(A')
# for general m, n
commutation(m, n) = Matrix(Permutation(vec(reshape(1:m*n, m, n)')))'
K23 = commutation(2, 3)
Commutation matrix allows us to exchange the two matrices in a Kronecker product. Let $\mathbf{A} \in \mathbb{R}^{m \times n}$, $\mathbf{B} \in \mathbb{R}^{p \times q}$, and $\mathbf{b} \in \mathbb{R}^{p}$. Then \begin{eqnarray*} \mathbf{K}_{pm} (\mathbf{A} \otimes \mathbf{B}) &=& (\mathbf{B} \otimes \mathbf{A}) \mathbf{K}_{qn} \\ \mathbf{K}_{pm} (\mathbf{A} \otimes \mathbf{B}) \mathbf{K}_{nq} &=& \mathbf{B} \otimes \mathbf{A} \\ \mathbf{K}_{pm} (\mathbf{A} \otimes \mathbf{b}) &=& \mathbf{b} \otimes \mathbf{A} \\ \mathbf{K}_{mp} (\mathbf{b} \otimes \mathbf{A}) &=& \mathbf{A} \otimes \mathbf{b}. \end{eqnarray*}
Commutation matrix allows us to transform the vec of a Kronecker product to the Kronecker product of the vecs. Let $\mathbf{A} \in \mathbb{R}^{m \times n}$ and $\mathbf{B} \in \mathbb{R}^{p \times q}$. Then $$ \text{vec} \, (\mathbf{A} \otimes \mathbf{B}) = (\mathbf{I}_n \otimes \mathbf{K}_{qm} \otimes \mathbf{I}_p) (\text{vec} \, \mathbf{A} \otimes \text{vec} \, \mathbf{B}). $$
Let $$ \mathbf{N}_n = \frac 12 (\mathbf{I}_{n^2} + \mathbf{K}_{nn}). $$ Then \begin{eqnarray*} \mathbf{N}_n &=& \mathbf{N}_n' = \mathbf{N}_n^2 \\ \text{rank}(\mathbf{N}_n) &=& \text{tr}(\mathbf{N}_n) = \frac{n(n+1)}{2} \\ \mathbf{N}_n \mathbf{K}_{nn} &=& \mathbf{N}_n = \mathbf{K}_{nn} \mathbf{N}_n. \end{eqnarray*}
Since the elements of $\text{vec} \, \mathbf{A}$ are those of $\text{vech} \, \mathbf{A}$ with some repetition, there exists an $n^2 \times \frac{n(n+1)}{2}$ duplication matrix $\mathbf{D}_n$ that tansforms, for symmetric $\mathbf{A}$, $\text{vech} \, \mathbf{A}$ to $\text{vec} \, \mathbf{A}$: $$ \mathbf{D}_n \text{vech} \, \mathbf{A} = \text{vec} \, \mathbf{A}. \quad \quad \quad (\mathbf{A} = \mathbf{A}'). $$
$\mathbf{D}_n$ has full column rank. Why?
The Moore-Penrose inverse of $\mathbf{D}_n$ is $$ \mathbf{D}_n^+ = (\mathbf{D}_n'\mathbf{D}_n)^{-1} \mathbf{D}_n'. $$ Thus $\text{vech} \, \mathbf{A}$ can be uniquely solved from $\text{vec} \, \mathbf{A}$ $$ \text{vech} \, \mathbf{A} = \mathbf{D}_n^+ \text{vec} \, \mathbf{A}. \quad \quad \quad (\mathbf{A} = \mathbf{A}'). $$
Some properties of $\mathbf{D}_n$. \begin{eqnarray*} \mathbf{K}_n \mathbf{D}_n &=& \mathbf{D}_n \\ \mathbf{D}_n \mathbf{D}_n^+ &=& \frac 12 (\mathbf{I}_{n^2} + \mathbf{K}_{nn}) \\ \mathbf{D}_n \mathbf{D}_n^+ (\mathbf{b} \otimes \mathbf{A}) &=& \frac 12 (\mathbf{b} \otimes \mathbf{A} + \mathbf{A} \otimes \mathbf{b}). \end{eqnarray*}
Let $\mathbf{A} \in \mathbb{R}^{n \times n}$. Then \begin{eqnarray*} \mathbf{D}_n' \text{vec} \, \mathbf{A} &=& \text{vech}(\mathbf{A} + \mathbf{A}' - \text{diag}(\mathbf{A})) \\ \text{det}(\mathbf{D}_n^+ (\mathbf{A} \otimes \mathbf{A}) \mathbf{D}_n^+) &=& 2^{-n(n-1)/2} \text{det}(\mathbf{A})^{n+1}. \end{eqnarray*}
n = 3
A = zeros(Int, n, n)
A[tril(trues(n, n))] = 1:6
LinearAlgebra.copytri!(A, 'L')
A
vech(A)
vec(A)
function duplication(n)
D = zeros(Int, abs2(n), n * (n + 1) >> 1)
vechidx = 1
for j in 1:n
for i in j:n
D[(j - 1) * n + i, vechidx] = 1
D[(i - 1) * n + j, vechidx] = 1
vechidx += 1
end
end
D
end
duplication(3)
Example at https://cs.stanford.edu/people/eroberts/courses/soco/projects/data-compression/lossy/jpeg/dct.htm.
n = 8
C = [cos((j + 0.5) * k * π / 8) for j in 0:n-1, k in 0:n-1]
The columns of DCT matrix $\mathbf{C}_n$ are orthogonal.
C'C
TODO
Example: Fisher iris data.