In this lecture we learn how to properly apply the chain rule to obtain derivatives of scalar, vector and matrix functions.
A reference I found extremely useful for this purpose is:
Below we collect formulas for the Jacobian matrices (first derivatives). A separate note collects those for the Hessian matrices (second derivatives).
| Scalar variable | Vector variable | Matrix variable | |
|---|---|---|---|
| Scalar function | $\phi(\xi)$ | $\phi(\mathbf{x})$ | $\phi(\mathbf{X})$ |
| Vector function | $f(\xi)$ | $f(\mathbf{x})$ | $f(\mathbf{X})$ |
| Matrix function | $F(\xi)$ | $F(\mathbf{x})$ | $F(\mathbf{X})$ |
For vector/matrix variables and vector/matrix functions, there is a question of how to organize all the partial derivatives in the Jacobian matrix. Following are a few popular notations for the Jacobian matrix. Unfortunately, none of them leads to a useful chain rule.
Defintion 1: For a differentiable scalar function: $\phi: \mathbb{R}^{n \times q} \mapsto \mathbb{R}$, $$ \frac{\partial \phi(\mathbf{X})}{\partial \mathbf{X}} = \begin{pmatrix} \frac{\partial \phi}{\partial x_{11}} & \cdots & \frac{\partial \phi}{\partial x_{1q}} \\ \vdots & \ddots & \vdots \\ \frac{\partial \phi}{\partial x_{n1}} & \cdots & \frac{\partial \phi}{\partial x_{nq}} \end{pmatrix}. $$
Defintion 2: For a matrix differentiable matrix function: $F = (f_{ij}): \mathbb{R}^{n \times q} \mapsto \mathbb{R}^{m \times p}$, $$ \frac{\partial F(\mathbf{X})}{\partial \mathbf{X}} = \begin{pmatrix} \frac{\partial f_{11}(\mathbf{X})}{\partial \mathbf{X}} & \cdots & \frac{\partial f_{1p}(\mathbf{X})}{\partial \mathbf{X}} \\ \vdots & \ddots & \vdots \\ \frac{\partial f_{m1}(\mathbf{X})}{\partial \mathbf{X}} & \cdots & \frac{\partial f_{mp}(\mathbf{X})}{\partial \mathbf{X}} \end{pmatrix}. $$
Definition 3: For a matrix differentiable matrix function: $F = (f_{ij}): \mathbb{R}^{n \times q} \mapsto \mathbb{R}^{m \times p}$, $$ \genfrac{}{}{3pt}{0}{\partial F(\mathbf{X})}{\partial \mathbf{X}} = \begin{pmatrix} \frac{\partial F(\mathbf{X})}{\partial x_{11}} & \cdots & \frac{\partial F(\mathbf{X})}{\partial x_{1p}} \\ \vdots & \ddots & \vdots \\ \frac{\partial F(\mathbf{X})}{\partial x_{m1}} & \cdots & \frac{\partial F(\mathbf{X})}{\partial x_{mp}} \end{pmatrix}. $$
Consider the example $F(\mathbf{X}) = \mathbf{X}$. $$ \frac{\partial F(\mathbf{X})}{\partial \mathbf{X}} = \genfrac{}{}{3pt}{0}{\partial F(\mathbf{X})}{\partial \mathbf{X}} = (\text{vec} \, \mathbf{I}_n)(\text{vec} \, \mathbf{I}_q)' $$ is even not the identity matrix $\mathbf{I}_{nq}$!
For a matrix differentiable matrix function: $F: \mathbb{R}^{n \times q} \mapsto \mathbb{R}^{m \times p}$, the Jacobian matrix of $F$ is the $mp \times nq$ matrix $$ \operatorname{D} F(\mathbf{X}) = \frac{\partial \text{vec} F(\mathbf{X})}{\partial (\text{vec} \mathbf{X})'}. $$ It includes the scalar function ($m = p = 1$) and vector function ($p=1$) as special cases.
Example. For $F(\mathbf{X}) = \mathbf{X}$, $$ \operatorname{D} F(\mathbf{X}) = \mathbf{I}_{nq}. $$
For a scalar function $\phi$, we have \begin{eqnarray*} \operatorname{D} \phi(\mathbf{x}) &=& \nabla \phi(\mathbf{x})' \\ \operatorname{D} \phi(\mathbf{X}) &=& \left( \text{vec} \, \frac{\partial \phi(\mathbf{X})}{\partial \mathbf{X}} \right)'. \end{eqnarray*}
For any function composition $$ h(\mathbf{X}) = g(f(\mathbf{X})), $$ or $$ \mathbf{X} \overset{f}{\mapsto} \mathbf{Y} \overset{g}{\mapsto} \mathbf{Z}, $$ where $\mathbf{X}$, $\mathbf{Y}$, and $\mathbf{Z}$ can have abitrary shapes (scalar, vector, matrix), we have $$ \operatorname{D} h(\mathbf{X}) = (\operatorname{D} g(\mathbf{Y})) (\operatorname{D} f(\mathbf{X})), $$ or $$ \operatorname{D}_{\mathbf{X}} \mathbf{Z} = \operatorname{D}_{\mathbf{Y}} \mathbf{Z} \cdot \operatorname{D}_{\mathbf{X}} \mathbf{Y}. $$
If $\phi(\mathbf{x}) = \mathbf{a}' \mathbf{x}$, then $$ \operatorname{D} \phi(\mathbf{x}) = \mathbf{a}'. $$
If $\phi(\mathbf{x}) = \mathbf{x}' \mathbf{A} \mathbf{x}$, then $$ \operatorname{D} \phi(\mathbf{x}) = \mathbf{x}' (\mathbf{A} + \mathbf{A}'). $$
If $\phi(\mathbf{x}) = \mathbf{a}' f(\mathbf{x})$, then $$ \operatorname{D} \phi(\mathbf{x}) = \mathbf{a}' \operatorname{D} f(\mathbf{x}). $$
If $\phi(\mathbf{x}) = f(\mathbf{x})' \mathbf{A} g(\mathbf{x})$, then $$ \operatorname{D} \phi(\mathbf{x}) = g(\mathbf{x})' \mathbf{A}' \operatorname{D} f(\mathbf{x}) + f(\mathbf{x})' \mathbf{A} \operatorname{D} g(\mathbf{x}). $$
If $\phi(\mathbf{x}) = \mathbf{x}_1' \mathbf{A} \mathbf{x}_2$, where $\mathbf{x} = \begin{pmatrix} \mathbf{x}_1 \\ \mathbf{x}_2 \end{pmatrix}$, then \begin{eqnarray*} \operatorname{D}_1 \phi(\mathbf{x}) &=& \mathbf{x}_2' \mathbf{A}' \\ \operatorname{D}_2 \phi(\mathbf{x}) &=& \mathbf{x}_1' \mathbf{A} \\ \operatorname{D} \phi(\mathbf{x}) &=& \mathbf{x}' \begin{pmatrix} \mathbf{O} & \mathbf{A} \\ \mathbf{A}' & \mathbf{O} \end{pmatrix}. \end{eqnarray*}
Following list for scalar functions gives formulas for $\frac{\partial \phi(\mathbf{X})}{\partial \mathbf{X}}$. To obtain the Jacobian matrix (a single row), apply the formula $$ \operatorname{D} \phi(\mathbf{X}) = \left( \text{vec} \, \frac{\partial \phi(\mathbf{X})}{\partial \mathbf{X}} \right)'. $$
If $\phi(\mathbf{X}) = \text{tr}(\mathbf{X})$, then $$ \frac{\partial \phi(\mathbf{X})}{\partial \mathbf{X}} = \mathbf{I} $$
If $\phi(\mathbf{X}) = \text{tr}(\mathbf{X}'\mathbf{X})$, then $$ \frac{\partial \phi(\mathbf{X})}{\partial \mathbf{X}} = 2 \mathbf{X}. $$
If $\phi(\mathbf{X}) = \text{tr}(\mathbf{X}^2)$ for a square matrix $\mathbf{X}$, then $$ \frac{\partial \phi(\mathbf{X})}{\partial \mathbf{X}} = 2 \mathbf{X}'. $$
If $\phi(\mathbf{X}) = \text{tr}(\mathbf{A} \mathbf{X})$, then $$ \frac{\partial \phi(\mathbf{X})}{\partial \mathbf{X}} = \mathbf{A}'. $$
If $\phi(\mathbf{X}) = \text{tr}(\mathbf{X} \mathbf{A} \mathbf{X}' \mathbf{B})$, then $$ \frac{\partial \phi(\mathbf{X})}{\partial \mathbf{X}} = \mathbf{B}' \mathbf{X} \mathbf{A}' + \mathbf{B} \mathbf{X} \mathbf{A}. $$
If $\phi(\mathbf{X}) = \text{tr}(\mathbf{X} \mathbf{A} \mathbf{X} \mathbf{B})$, then $$ \frac{\partial \phi(\mathbf{X})}{\partial \mathbf{X}} = \mathbf{B}' \mathbf{X}' \mathbf{A}' + \mathbf{A}' \mathbf{X}' \mathbf{B}'. $$
If $\phi(\mathbf{X}) = \text{tr}(\mathbf{A} \mathbf{X}^{-1})$, then $$ \frac{\partial \phi(\mathbf{X})}{\partial \mathbf{X}} = - (\mathbf{X}^{-1} \mathbf{A} \mathbf{X}^{-1})'. $$
If $\phi(\mathbf{X}) = \text{tr}(\mathbf{X}^p)$ for a square matrix, then $$ \frac{\partial \phi(\mathbf{X})}{\partial \mathbf{X}} = p (\mathbf{X}')^{p-1}, \quad p=1,2,... $$
If $\phi(\mathbf{X}) = \text{tr} F(\mathbf{X})$, then $$ \operatorname{D} \phi(\mathbf{X}) = (\text{vec} \, \mathbf{I})' \operatorname{D} F(\mathbf{X}). $$
If $\phi(\mathbf{X}, \mathbf{Z}) = \text{tr}(\mathbf{A} \mathbf{X} \mathbf{B} \mathbf{Z})$, then \begin{eqnarray*} \frac{\partial \phi(\mathbf{X}, \mathbf{Z})}{\partial \mathbf{X}} &=& \mathbf{A}' \mathbf{Z}' \mathbf{B}' \\ \frac{\partial \phi(\mathbf{X}, \mathbf{Z})}{\partial \mathbf{Z}} &=& \mathbf{B}' \mathbf{X}' \mathbf{A}' \\ \operatorname{D}_\mathbf{X} \phi(\mathbf{X}, \mathbf{Z}) &=& (\text{vec} \, \mathbf{A}' \mathbf{Z}' \mathbf{B}')' \\ \operatorname{D}_\mathbf{Z} \phi(\mathbf{X}, \mathbf{Z}) &=& (\text{vec} \, \mathbf{B}' \mathbf{X}' \mathbf{A}')' \\ \operatorname{D} \phi(\mathbf{X}, \mathbf{Z}) &=& (\operatorname{D}_\mathbf{X} \phi(\mathbf{X}, \mathbf{Z}), \operatorname{D}_\mathbf{Z} \phi(\mathbf{X}, \mathbf{Z})). \end{eqnarray*}
If $\phi(\mathbf{X}) = \text{tr}(\mathbf{A} F(\mathbf{X}) \mathbf{B} G(\mathbf{X}))$, then $$ \operatorname{D} \phi(\mathbf{X}) = (\text{vec} \, \mathbf{A}' G(\mathbf{X})' \mathbf{B}')' \operatorname{D} F(\mathbf{X}) + (\text{vec} \, \mathbf{B}' F(\mathbf{X})' \mathbf{A}')' \operatorname{D} G(\mathbf{X}). $$
If $\phi(\mathbf{X}) = \text{det} \, \mathbf{X}$ for a non-singular matrix, then $$ \frac{\partial \phi(\mathbf{X})}{\partial \mathbf{X}} = (\text{det} \, \mathbf{X}) (\mathbf{X}')^{-1}. $$
If $\phi(\mathbf{X}) = \text{det} \, \mathbf{A} \mathbf{X} \mathbf{B}$, then $$ \frac{\partial \phi(\mathbf{X})}{\partial \mathbf{X}} = (\text{det} \, \mathbf{A} \mathbf{X} \mathbf{B}) \mathbf{A}' (\mathbf{B}' \mathbf{X}' \mathbf{A}')^{-1} \mathbf{B}' $$ if the inverse exists.
If $\phi(\mathbf{X}) = \text{det} \, \mathbf{X} \mathbf{X}'$, then $$ \frac{\partial \phi(\mathbf{X})}{\partial \mathbf{X}} = 2 (\text{det} \, \mathbf{X} \mathbf{X}') (\mathbf{X}\mathbf{X}')^{-1} \mathbf{X}. $$
If $\phi(\mathbf{X}) = \text{det} \, \mathbf{X}'\mathbf{X}$, then $$ \frac{\partial \phi(\mathbf{X})}{\partial \mathbf{X}} = 2 (\text{det} \, \mathbf{X}'\mathbf{X}) \mathbf{X} (\mathbf{X}'\mathbf{X})^{-1}. $$
If $\phi(\mathbf{X}) = \log \text{det} \, \mathbf{X}' \mathbf{A} \mathbf{X}$ where $\mathbf{A}$ is positive definite and $\mathbf{X}$ has full column rank, then $$ \operatorname{D} \phi(\mathbf{X}) = (\text{vec} \, (\mathbf{X}' \mathbf{A} \mathbf{X})^{-1})' \operatorname{D} \mathbf{X}' \mathbf{A} \mathbf{X}. $$
If $\phi(\mathbf{X}) = \text{det} \, \mathbf{X}^2$ for a square matrix, then $$ \frac{\partial \phi(\mathbf{X})}{\partial \mathbf{X}} = 2 (\text{det} \, \mathbf{X})^2 (\mathbf{X}')^{-1}. $$
If $\phi(\mathbf{X}) = \text{det} \, \mathbf{X}^p$ for a square matrix, then $$ \frac{\partial \phi(\mathbf{X})}{\partial \mathbf{X}} = p (\text{det} \, \mathbf{X})^{p} (\mathbf{X}')^{-1} $$ for $p$ either a positive or a negative integer.
Let $F(\mathbf{X})$ be a square non-singular matrix function of $\mathbf{X}$, and $G(\mathbf{X}) = \mathbf{C} F(\mathbf{X})^{-1} \mathbf{A}$. Then $$ \frac{\partial \, \text{det} \, F(\mathbf{X})}{\partial \mathbf{X}} = \begin{cases} (\det F(\mathbf{X})) (\mathbf{G} \mathbf{X} \mathbf{B} + \mathbf{G}' \mathbf{X} \mathbf{B}') & \text{if } F(\mathbf{X}) = \mathbf{A} \mathbf{X} \mathbf{B} \mathbf{X}' \mathbf{C} \\ (\det F(\mathbf{X})) (\mathbf{B} \mathbf{X} \mathbf{G} + \mathbf{B}' \mathbf{X} \mathbf{G}') & \text{if } F(\mathbf{X}) = \mathbf{A} \mathbf{X}' \mathbf{B} \mathbf{X} \mathbf{C} \\ (\det F(\mathbf{X})) (\mathbf{G} \mathbf{X} \mathbf{B} + \mathbf{B} \mathbf{X} \mathbf{G})' & \text{if } F(\mathbf{X}) = \mathbf{A} \mathbf{X} \mathbf{B} \mathbf{X} \mathbf{C} \end{cases}. $$
If $f(\mathbf{x}) = \mathbf{A} \mathbf{x}$, then $$ \operatorname{D} f(\mathbf{x}) = \mathbf{A}. $$
If $f(\mathbf{x}) = \mathbf{A} g(\mathbf{x})$, then $$ \operatorname{D} f(\mathbf{x}) = \mathbf{A} \operatorname{D} g(\mathbf{x}). $$
If $f(\mathbf{X}) = \mathbf{X} \mathbf{a}$, then $$ \operatorname{D} f(\mathbf{X}) = \mathbf{a}' \otimes \mathbf{I}. $$
If $f(\mathbf{X}) = \mathbf{X}' \mathbf{a}$, then $$ \operatorname{D} f(\mathbf{X}) = \mathbf{I} \otimes \mathbf{a}'. $$
If $f(\mathbf{X}) = \operatorname{vec} \mathbf{X}$ where $\mathbf{X} \in \mathbb{R}^{n \times q}$, then $$ \operatorname{D} f(\mathbf{X}) = \mathbf{I}_{nq}. $$
If $f(\mathbf{X}) = \operatorname{vech} \mathbf{X}$, where $\mathbf{X} \in \mathbb{R}^{n \times n}$ is symmetric, then $$ \operatorname{D} f(\mathbf{X}) = \mathbf{D}_{n}^+ = (\mathbf{D}_{n}'\mathbf{D}_{n})^{-1} \mathbf{D}_{n}', $$ where $\mathbf{D}_n \in \mathbb{R}^{n^2 \times \frac{n(n+1)}{2}}$ is the Duplication matrix.
If $f(\mathbf{X}) = \operatorname{diag} \mathbf{X}$, where $\mathbf{X} \in \mathbb{R}^{n \times n}$, then $$ \operatorname{D} f(\mathbf{X}) = \begin{pmatrix} \mathbf{e}_1' \otimes \mathbf{e}_1' \\ \vdots \\ \mathbf{e}_n' \otimes \mathbf{e}_n' \end{pmatrix} = (\mathbf{I}_n \odot \mathbf{I}_n)', $$ where $\odot$ denotes the Khatri-Rao (column-wise Kronecker) product.
If $f(\mathbf{x}) = (\mathbf{x}' \mathbf{x}) \mathbf{a}$, then $$ \operatorname{D} f(\mathbf{x}) = 2 \mathbf{a} \mathbf{x}'. $$
If $f(\mathbf{x}) = (\mathbf{x}' \mathbf{x}) g(\mathbf{x})$, then $$ \operatorname{D} f(\mathbf{x}) = 2 g(\mathbf{x}) \mathbf{x}' \operatorname{D} g(\mathbf{x}). $$
Let $\lambda_0$ be a simple eigenvalue of $\mathbf{X}$ with normalized eigenvector $\mathbf{u}_0$. $$ \operatorname{D} u(\mathbf{X}) = \mathbf{u}_0' \otimes (\lambda_0 \mathbf{I}_n - \mathbf{X}_0)^+. $$
If $F(x) = x \cdot \mathbf{A} + \mathbf{B}$, then $$ \operatorname{D} F(x) = \operatorname{vec} \mathbf{A}. $$
If $F(\mathbf{X}) = $ the $n \times n$ symmetric matrix with lower triangular part determined from $\operatorname{vech} \mathbf{X}$, then $$ \operatorname{D} F(\mathbf{X}) = \mathbf{D}_{n}, $$ where $\mathbf{D}_{n} \in \mathbb{R}^{n^2 \times \frac{n(n+1)}{2}}$ is the Duplication matrix.
If $F(\mathbf{X}) = $ the $n \times n$ lower triangular matrix with lower triangular part determined from $\operatorname{vech} \mathbf{X}$, then $$ \operatorname{D} F(\mathbf{X}) = \mathbf{C}_{n}, $$ where $\mathbf{C}_{n} \in \mathbb{R}^{n^2 \times \frac{n(n+1)}{2}}$ is the copying matrix such that $\operatorname{vec} \mathbf{X} = \mathbf{C}_n \operatorname{vech} \mathbf{X}$.
If $F(\mathbf{X}) = \mathbf{X}$ where $\mathbf{X} \in \mathbb{R}^{n \times q}$, then $$ \operatorname{D} F(\mathbf{X}) = \mathbf{I}_{nq}. $$
If $F(\mathbf{X}) = \mathbf{X}'$ where $\mathbf{X} \in \mathbb{R}^{n \times q}$, then $$ \operatorname{D} F(\mathbf{X}) = \mathbf{K}_{nq}, $$ where $\mathbf{K}_{nq} \in \mathbb{R}^{nq \times nq}$ is the commutation matrix.
If $F(\mathbf{x}) = \operatorname{diag}(\mathbf{x})$, then $$ \operatorname{D} F(\mathbf{x}) = \mathbf{I} \odot \mathbf{I}, $$ where $\odot$ is the Khatri-Rao (columnwide Kronecker) product.
If $F(\mathbf{X})= \mathbf{A} \mathbf{X} \mathbf{B}$, then $$ \operatorname{D} F(\mathbf{X}) = \mathbf{B}' \otimes \mathbf{A}. $$
If $F(\mathbf{x}) = \mathbf{x} \mathbf{x}'$, then $$ \operatorname{D} F(\mathbf{x}) = \mathbf{I} \otimes \mathbf{x} + \mathbf{x} \otimes \mathbf{I}. $$
If $F(\mathbf{X}) = \mathbf{X} \mathbf{X}'$ where $\mathbf{X} \in \mathbb{R}^{n \times q}$, then $$ \operatorname{D} F(\mathbf{X}) = 2 \mathbf{N}_{n} (\mathbf{X} \otimes \mathbf{I}_n), $$ where $$ \mathbf{N}_{n} = \frac 12 (\mathbf{I}_{n^2} + \mathbf{K}_{nn}). $$
If $F(\mathbf{X}) = \mathbf{X}' \mathbf{X}$ where $\mathbf{X} \in \mathbb{R}^{n \times q}$, then $$ \operatorname{D} F(\mathbf{X}) = 2 \mathbf{N}_{q} (\mathbf{I}_q \otimes \mathbf{X}'), $$ where $$ \mathbf{N}_{q} = \frac 12 (\mathbf{I}_{q^2} + \mathbf{K}_{qq}). $$
If $F(\mathbf{X}) = \mathbf{X}^{-1}$, where $\mathbf{X} \in \mathbb{R}^{n \times n}$ is nonsingular, then $$ \operatorname{D} F(\mathbf{X}) = - (\mathbf{X}')^{-1} \otimes \mathbf{X}^{-1}. $$
If $F(\mathbf{X}) = \mathbf{X}^p$ where $\mathbf{X}$ is a square matrix and $p \ge 1$, then $$ \operatorname{D} F(\mathbf{X}) = \sum_{j=1}^p (\mathbf{X}')^{p-j} \otimes \mathbf{X}^{j-1}. $$
If $F(\mathbf{X}) = \mathbf{A} \mathbf{X}^{-1} \mathbf{B}$ where $\mathbf{X}$ is square and nonsingular, then $$ \operatorname{D} F(\mathbf{X}) = - \mathbf{B}' \mathbf{X}'^{-1} \otimes \mathbf{A} \mathbf{X}^{-1}. $$
For $\mathbf{X} \in \mathbb{R}^{n \times q}$, \begin{eqnarray*} \operatorname{D} \mathbf{X} \mathbf{A} \mathbf{X}' &=& \mathbf{X} \mathbf{A}' \otimes \mathbf{I}_n + (\mathbf{I}_n \otimes \mathbf{X} \mathbf{A}) \mathbf{K}_{nq} \\ \operatorname{D} \mathbf{X}' \mathbf{A} \mathbf{X} &=& (\mathbf{X}' \mathbf{A}' \otimes \mathbf{I}_q) \mathbf{K}_{nq} + \mathbf{I}_q \otimes \mathbf{X}' \mathbf{A} \\ \operatorname{D} \mathbf{X} \mathbf{A} \mathbf{X} &=& \mathbf{X}' \mathbf{A}' \otimes \mathbf{I}_n + \mathbf{I}_q \otimes \mathbf{X} \mathbf{A} \\ \operatorname{D} \mathbf{X}' \mathbf{A} \mathbf{X}' &=& (\mathbf{X} \mathbf{A}' \otimes \mathbf{I}_q) \mathbf{K}_{nq} + (\mathbf{I}_n \otimes \mathbf{X}' \mathbf{A}) \mathbf{K}_{nq}. \end{eqnarray*}
Moore-Penrose inverse: $$ \operatorname{D} \mathbf{X}^+ = - \mathbf{X}^+ \otimes \mathbf{X}^+ + (\mathbf{I} - \mathbf{X} \mathbf{X}^+) \otimes \mathbf{X}^+ (\mathbf{X}^{+})' \mathbf{K}_{nq} + (\mathbf{X}^+)' \mathbf{X}^+ \otimes (\mathbf{I} - \mathbf{X}^+ \mathbf{X}) \mathbf{K}_{nq}. $$
Adjoint matrix: $$ \operatorname{D} \mathbf{F}^\# = (\det \mathbf{F}) [(\text{vec} \, \mathbf{F}^{-1})(\text{vec} \, (\mathbf{F}')^{-1})' - (\mathbf{F}')^{-1} \otimes \mathbf{F}^{-1}]. $$
If $F(\mathbf{X}) = \mathbf{A} G(\mathbf{X}) \mathbf{B} H(\mathbf{X}) \mathbf{C}$, then $$ \operatorname{D} F(\mathbf{X}) = (\mathbf{C}' H(\mathbf{X})' \mathbf{B}' \otimes \mathbf{A}) \operatorname{D} G(\mathbf{X}) + (\mathbf{C}' \otimes \mathbf{A} G(\mathbf{X}) \mathbf{B}) \operatorname{D} H(\mathbf{X}). $$
Product rule. Let $F(\mathbf{X}) \in \mathbb{R}^{m \times p}$ and $G(\mathbf{X}) \in \mathbb{R}^{p \times r}$ be functions of $\mathbf{X} \in \mathbb{R}^{n \times q}$. Then $$ \operatorname{D} F(\mathbf{X})G(\mathbf{X}) = (G(\mathbf{X})' \otimes \mathbf{I}_m) \operatorname{D} F(\mathbf{X}) + (\mathbf{I}_r \otimes F(\mathbf{X}) \operatorname{D} G(\mathbf{X}). $$
Kronecker product: If $F(\mathbf{X}, \mathbf{Y}) = \mathbf{X} \otimes \mathbf{Y}$, where $\mathbf{X} \in \mathbb{R}^{n \times q}$ and $\mathbf{Y} \in \mathbb{R}^{p \times r}$, then $$ \operatorname{D} F(\mathbf{X}, \mathbf{Y}) = (\mathbf{I}_q \otimes \mathbf{K}_{rn} \otimes \mathbf{I}_p)(\mathbf{I}_{nq} \times \text{vec} \, \mathbf{Y} : \text{vec} \, \mathbf{X} \otimes \mathbf{I}_{pr}). $$
If $F(\mathbf{X}, \mathbf{Y}) = \mathbf{X} \mathbf{X}' \otimes \mathbf{Y} \mathbf{Y}'$, where $\mathbf{X}$ has $n$ rows and $\mathbf{Y}$ has $p$ rows, then $$ \operatorname{D} F(\mathbf{X}, \mathbf{Y}) = (\mathbf{I}_n \otimes \mathbf{K}_{pn} \otimes \mathbf{I}_p) [G_n(\mathbf{X}) \otimes \operatorname{vec} \mathbf{Y} \mathbf{Y}' , \operatorname{vec} \mathbf{X} \mathbf{X}' \otimes G_p(\mathbf{Y})], $$ where $$ G_m(\mathbf{A}) = (\mathbf{I}_{m^2} + \mathbf{K}_{mm}) (\mathbf{A} \otimes \mathbf{I}_m) $$ for any matrix $\mathbf{A}$ having $m$ rows.
Hadamard (elementwise) product: If $F(\mathbf{X}) = \mathbf{A} \odot \mathbf{X} = \mathbf{X} \odot \mathbf{A}$, then $$ \operatorname{D} F(\mathbf{X}) = \operatorname{diag} (\operatorname{vec} \mathbf{A}). $$
Hadamard (elementwise) product: If $F(\mathbf{X}, \mathbf{Y}) = \mathbf{X} \odot \mathbf{Y}$, then $$ \operatorname{D} F(\mathbf{X}, \mathbf{Y}) = (\operatorname{diag} (\operatorname{vec} \mathbf{Y}) : \operatorname{diag} (\operatorname{vec} \mathbf{X})). $$