关于矩阵求导（其一）

概念准备

在进行矩阵求导之前，我们首先需要做一些关于概念上的铺垫。主要分为以下两个部分：

函数与变元的种类
分子布局与分母布局

函数与变元的种类

这里的种类指的就是标量、向量和矩阵这三种。在本文后续内容中对于这三种形式的符号表达都遵循如下定义:

标量，用正文字体小写字母表示，例如： $x, y$
向量，用粗体小写字母表示，例如： $\mathbf{x}_{n \times 1} = \begin{pmatrix} x_1, x_2, \cdots, x_n \end{pmatrix}^T$
矩阵，用粗体大写字母表示，例如： $\mathbf{X}_{m \times n} = \begin{pmatrix} x_{11} & x_{12} & \cdots & x_{1n} \\ x_{21} & x_{22} & \cdots & x_{2n} \\ \vdots & \vdots & \ddots & \vdots \\ x_{m1} & x_{m2} & \cdots & x_{mn} \end{pmatrix}$

接下来是具体分类：

函数为标量，称之为实值标量函数，根据变元种类分为以下三种：
- 变元为标量，例:
\[f(x) = x^2\]
- 变元为向量，例:
\[\begin{aligned} & \mathbf{x}_{3 \times 1} = \begin{pmatrix} x_1, x_2, x_3 \end{pmatrix}^T \\[5mm] & f(\mathbf{x}) = x_1^2 + x_2^2 + x_3^2 \end{aligned}\]
- 变元为矩阵，例：
\[\begin{aligned} \mathbf{X}_{2 \times 3} &= \begin{pmatrix} x_{11} & x_{12} & x_{13} \\ x_{21} & x_{22} & x_{23} \end{pmatrix} \\[5mm] f(\mathbf{X}) &= x_{11}^2 + x_{12} + x_{13} + x_{21} + x_{22} + x_{23} \end{aligned}\]
函数为向量，称之为实向量函数，根据变元种类分为以下三种：
- 变元为标量，例：
\[\mathbf{f}_{3 \times 1}(x) = \begin{pmatrix} f_1(x) \\ f_2(x) \\ f_3(x) \end{pmatrix} = \begin{pmatrix} x \\ x^2 \\ x^3 \end{pmatrix}\]
- 变元为向量，例：
\[\begin{aligned} \mathbf{x}_{3 \times 1} &= \begin{pmatrix} x_1 & x_2 & x_3 \end{pmatrix}^T \\[5mm] \mathbf{f}_{3 \times 1}(\mathbf{x}) &= \begin{pmatrix} f_1(\mathbf{x}) \\ f_2(\mathbf{x}) \\ f_3(\mathbf{x}) \end{pmatrix} = \begin{pmatrix} x_1 + x_2 \\ x_2 + x_3 \\ x_3 + x_1 \end{pmatrix} \end{aligned}\]
- 变元为矩阵，例：
\[\begin{aligned} \mathbf{X}_{2 \times 3} &= \begin{pmatrix} x_{11} & x_{12} & x_{13} \\ x_{21} & x_{22} & x_{23} \end{pmatrix} \\[7mm] \mathbf{f}_{3 \times 1}(\mathbf{X}) &= \begin{pmatrix} f_1(\mathbf{\mathbf{X}}) \\ f_2(\mathbf{\mathbf{X}}) \\ f_3(\mathbf{\mathbf{X}}) \end{pmatrix} \\[7mm] &= \begin{pmatrix} x_{11} + x_{12} + x_{21}^2 \\ x_{13} + x_{21} + x_{22}^2 \\ x_{22} + x_{23} + x_{23}^2 \end{pmatrix} \end{aligned}\]
函数为矩阵，称之为实矩阵函数，根据变元种类分为以下三种：
- 变元为标量，例：
\[\begin{aligned} \mathbf{F}_{2 \times 3}(x) &= \begin{pmatrix} f_{11}(x) & f_{12}(x) & f_{13}(x) \\ f_{21}(x) & f_{22}(x) & f_{23}(x) \end{pmatrix} \\[5mm] &= \begin{pmatrix} x & 2x & 3x \\ x^2 & 2x^2 & 3x^3 \end{pmatrix} \end{aligned}\]
- 变元为向量，例：
\[\begin{aligned} \mathbf{x}_{3 \times 1} &= \begin{pmatrix} x_{11} & x_{12} & x_{13} \end{pmatrix}^T \\[5mm] \mathbf{F}_{2 \times 3}(\mathbf{x}) &= \begin{pmatrix} f_{11}(\mathbf{x}) & f_{12}(\mathbf{x}) & f_{13}(\mathbf{x}) \\ f_{21}(\mathbf{x}) & f_{22}(\mathbf{x}) & f_{23}(\mathbf{x}) \end{pmatrix} \\[5mm] &= \begin{pmatrix} x_{11} + x_{12} + x_{13} & 2x_{11} + x_{12} + x_{13} & 3x_{11} + x_{12} + x_{13} \\ 4x_{11} + x_{12} + x_{13} & 5x_{11} + x_{12} + x_{13} & 6x_{11} + x_{12} + x_{13} \end{pmatrix} \end{aligned}\]
- 变元为矩阵，例：
\[\begin{aligned} \mathbf{X}_{2 \times 3} &= \begin{pmatrix} x_{11} & x_{12} & x_{13} \\ x_{21} & x_{22} & x_{23} \end{pmatrix} \\[7mm] \mathbf{F}_{3 \times 2}(\mathbf{X}) &= \begin{pmatrix} f_{11}(\mathbf{X}) & f_{12}(\mathbf{X}) \\ f_{21}(\mathbf{X}) & f_{22}(\mathbf{X}) \\ f_{31}(\mathbf{X}) & f_{32}(\mathbf{X}) \end{pmatrix} \\[7mm] &= \begin{pmatrix} x_{11} + x_{12} + x_{13} + x_{21} + x_{22} + x_{23} & 2x_{11} + x_{12} + x_{13} + x_{21} + x_{22} + x_{23} \\ 3x_{11} + x_{12} + x_{13} + x_{21} + x_{22} + x_{23} & 4x_{11} + x_{12} + x_{13} + x_{21} + x_{22} + x_{23} \\ 5x_{11} + x_{12} + x_{13} + x_{21} + x_{22} + x_{23} & 6x_{11} + x_{12} + x_{13} + x_{21} + x_{22} + x_{23} \end{pmatrix} \end{aligned}\]

最后关于函数和变元的种类，可以总结如下：

函数 \ 变元	标量变元	向量变元	矩阵变元
实值标量函数	$f(x)$	$f(\mathbf{x})$	$f\mathbf{(X)}$
实向量函数	$\mathbf{f}(x)$	$\mathbf{f}(\mathbf{x})$	$\mathbf{f}(\mathbf{X})$
实矩阵函数	$\mathbf{F}(x)$	$\mathbf{F}(\mathbf{x}) $	$\mathbf{F}(\mathbf{X})$

分子布局与分母布局

进行矩阵求导的时候实际上是将函数中的每一个分量依次对变元中的每一个分量求偏导，并将所有求偏导的结果排列起来，根据不同的排列规则分为分子布局 (Numerator Layout) 和 分母布局 (Denominator Layout)。举一个例子：

\[\mathbf{x} = \begin{pmatrix} x_1 \\ x_2 \\ \vdots \\ x_n \end{pmatrix}_{n \times 1} , \qquad \frac{\partial{f}}{\partial{\mathbf{x}}} = \begin{pmatrix} \frac{\partial{f}}{\partial{x_1}} \\ \frac{\partial{f}}{\partial{x_2}} \\ \vdots \\ \frac{\partial{f}}{\partial{x_n}} \end{pmatrix}_{n \times 1}\]

这里的 $f(\mathbf{x})$ 是一个实值标量函数，而变元 $\mathbf{x}$ 是一个 $n \times 1$ 的列向量，求完导后的结果也是一个 $n \times 1$ 的列向量。如果把这里求导的式子 $\frac{\partial{f}}{\partial{\mathbf{x}}}$ 视作分式的话可以发现，求导结果的维度与分母 $\mathbf{x}$ 一致，所以这里采用的是分母布局。再举一个例子：

\[\mathbf{x} = \begin{pmatrix} x_1 \\ x_2 \\ \vdots \\ x_n \end{pmatrix}_{n \times 1} , \qquad \mathbf{f} = \begin{pmatrix} f_1 \\ f_2 \\ \vdots \\ f_m \end{pmatrix}_{m \times 1} , \qquad \frac{\partial{\mathbf{f}}}{\partial{\mathbf{x}}} = \large \begin{pmatrix} \frac{\partial{\color{royalblue}f_1}}{\partial{\color{orange}x_1}} & \frac{\partial{\color{royalblue}{f_2}}}{\partial{x_1}} & \color{royalblue}{\cdots} & \frac{\partial{\color{royalblue}{f_m}}}{\partial{x_1}} \\ \frac{\partial{f_1}}{\partial{\color{orange}{x_2}}} & \frac{\partial{f_2}}{\partial{x_2}} & \cdots & \frac{\partial{f_m}}{\partial{x_2}} \\ \color{orange}{\vdots} & \vdots & \ddots & \vdots \\ \frac{\partial{f_1}}{\partial{\color{orange}{x_n}}} & \frac{\partial{f_2}}{\partial{x_n}} & \cdots & \frac{\partial{f_m}}{\partial{x_n}} \end{pmatrix}_{\color{orange}{n} \times \color{royalblue}{m}}\]

这里的 $\mathbf{f}(\mathbf{x})$ 是一个 $m \times 1$ 的实向量函数，而变元 $\mathbf{x}$ 是一个 $n \times 1$ 的列向量，求导的结果是一个 $\color{orange}{n} \times \color{royalblue}{m}$ 的矩阵。我们可以看到这个结果符合变元 $\mathbf{x}$ 的排列方式（橙色部分），而函数 $\mathbf{f}$ 则转变为行向量的排列方式（蓝色部分），也就是转置了。这里采用的还是分母布局。如果采用分子布局的话：

\[\frac{\partial{\mathbf{f}}}{\partial{\mathbf{x}}} = \large \begin{pmatrix} \frac{\partial{\color{royalblue}{f_1}}}{\partial{\color{orange}{x_1}}} & \frac{\partial{f_1}}{\partial{\color{orange}{x_2}}} & \color{orange}{\cdots} & \frac{\partial{f_1}}{\partial{\color{orange}{x_n}}} \\ \frac{\partial{\color{royalblue}{f_2}}}{\partial{x_1}} & \frac{\partial{f_2}}{\partial{x_2}} & \cdots & \frac{\partial{f_2}}{\partial{x_n}} \\ \color{royalblue}{\vdots} & \vdots & \ddots & \vdots \\ \frac{\partial{\color{royalblue}{f_m}}}{\partial{x_1}} & \frac{\partial{f_m}}{\partial{x_2}} & \cdots & \frac{\partial{f_m}}{\partial{x_n}} \end{pmatrix}_{\color{royalblue}{m} \times \color{orange}{n}}\]

结果就是一个 $\color{royalblue}{m} \times \color{orange}{n}$ 的矩阵，变元 $\mathbf{x}$ 发生了转置。

函数 \ 变元	标量变元	向量变元	矩阵变元
实值标量函数	$f(x)$	$\color{red}{f(\mathbf{x})}$	$\color{red}{f\mathbf{(X)}}$
实向量函数	$\color{red}{\mathbf{f}(x)}$	$\color{red}{\mathbf{f}(\mathbf{x})}$	$\mathbf{f}(\mathbf{X})$
实矩阵函数	$\color{red}{\mathbf{F}(x)}$	$\mathbf{F}(\mathbf{x}) $	$\mathbf{F}(\mathbf{X})$

本文后续关于求导的部分只讨论上表中标注为红色的情况，那么关于求导的布局可以通俗地总结为：矩阵求导结果是什么布局由分子、分母中没有发生转置的那一个决定。

矩阵求导公式

以下表中所有的向量初始状态都默认为列向量。例如：

\[\mathbf{x} = \begin{pmatrix} x_1 \\ \vdots \\ x_n \end{pmatrix}_{n \times 1}\]

向量对向量 (Vector-by-vector)

Codition	Expression	Numerator layout	Denominator layout	Proof
$\large \mathbf{a}$ is not a function of $\large \mathbf{x}$	$\Large \frac{\partial{\mathbf{a}}}{\partial{\mathbf{x}}}$	$\large \mathbf{0}$	$\large \mathbf{0}$	Proof 1-1
	$\Large \frac{\partial{\mathbf{x}}}{\partial{\mathbf{x}}}$	$\large \mathbf{I}$	$\large \mathbf{I}$
$\large \mathbf{A}$ is not a funtion of $\large \mathbf{x}$	$\Large \frac{\partial{\mathbf{Ax}}}{\partial{\mathbf{x}}}$	$\large \mathbf{A}$	$\large \mathbf{A}^T$	Proof 1-2
$\large \mathbf{A}$ is not a funtion of $\large \mathbf{x}$	$\Large \frac{\partial{\mathbf{x}^T\mathbf{A}}}{\partial{\mathbf{x}}}$	$\large \mathbf{A}^T$	$\large \mathbf{A}$	Proof 1-3
$\large a$ is not a function of $\large \mathbf{x}$, $\mathbf{u} = \mathbf{u}(\mathbf{x})$	$\Large \frac{\partial{a\mathbf{u}}}{\partial{\mathbf{x}}}$	$a$ $\large \frac{\partial{\mathbf{u}}}{\partial{\mathbf{x}}}$	$a$ $\large \frac{\partial{\mathbf{u}}}{\partial{\mathbf{x}}}$	Proof 1-4
$v = v(\mathbf{x}), \; \mathbf{u} = \mathbf{u}(\mathbf{x})$	$\Large \frac{\partial{v \mathbf{u}}}{\partial{\mathbf{x}}}$	$v$ $\large \frac{\partial{\mathbf{u}}}{\partial{\mathbf{x}}} +$ $\mathbf{u}$ $\large \frac{\partial{v}}{\partial{\mathbf{x}}}$	$v$ $\large \frac{\partial{\mathbf{u}}}{\partial{\mathbf{x}}} + \frac{\partial{v}}{\partial{\mathbf{x}}}$ $\mathbf{u}^T$	Proof 1-5
$\large \mathbf{A}$ is not a function of $\large \mathbf{x}$, $\mathbf{u} = \mathbf{u}(\mathbf{x})$	$\Large \frac{\partial{\mathbf{Au}}}{\partial{\mathbf{x}}}$	$\mathbf{A}$ $\large \frac{\partial{\mathbf{u}}}{\partial{\mathbf{x}}}$	$\large \frac{\partial{\mathbf{u}}}{\partial{\mathbf{x}}}$ $\mathbf{A}^T$	Proof 1-6
$\mathbf{u} = \mathbf{u}(\mathbf{x}), \; \mathbf{v} = \mathbf{v}(\mathbf{x})$	$\Large \frac{\partial{(\mathbf{u} + \mathbf{v})}}{\partial{\mathbf{x}}}$	$\large \frac{\partial{\mathbf{u}}}{\partial{\mathbf{x}}} + \frac{\partial{\mathbf{v}}}{\partial{\mathbf{x}}}$	$\large \frac{\partial{\mathbf{u}}}{\partial{\mathbf{x}}} + \frac{\partial{\mathbf{v}}}{\partial{\mathbf{x}}}$
$\mathbf{u} = \mathbf{u}(\mathbf{x})$	$\Large \frac{\partial{\mathbf{g}(\mathbf{u})}}{\partial{\mathbf{x}}}$	$\large \frac{\partial{\mathbf{g}(\mathbf{u})}}{\partial{\mathbf{u}}} \frac{\partial{\mathbf{u}}}{\partial{\mathbf{x}}}$	$\large \frac{\partial{\mathbf{u}}}{\partial{\mathbf{x}}} \frac{\partial{\mathbf{g}(\mathbf{u})}}{\partial{\mathbf{x}}}$	Proof 1-7
$\mathbf{u} = \mathbf{u}(\mathbf{x})$	$\Large \frac{\partial{\mathbf{f}(\mathbf{g}(\mathbf{u}))}}{\partial{\mathbf{x}}}$	$\large \frac{\partial{\mathbf{f}(\mathbf{g})}}{\partial{\mathbf{g}}} \frac{\partial{\mathbf{g}(\mathbf{u})}}{\partial{\mathbf{u}}} \frac{\partial{\mathbf{u}}}{\partial{\mathbf{x}}}$	$\large \frac{\partial{\mathbf{u}}}{\partial{\mathbf{x}}} \frac{\partial{\mathbf{g}(\mathbf{u})}}{\partial{\mathbf{u}}} \frac{\partial{\mathbf{f}(\mathbf{g})}}{\partial{\mathbf{g}}}$	Proof 1-8

标量对向量 (Scalar-by-vector)

Codition	Expression	Numerator layout (row vector)	Denominator layout (column vector)	Proof
$\large a$ is not a function of $\large \mathbf{x}$	$\Large \frac{\partial{a}}{\partial{\mathbf{x}}}$	$\large \mathbf{0}^T$	$\large \mathbf{0}$
$\large a$ is not a function of $\large \mathbf{x}$, $u = u(\mathbf{x})$	$\Large \frac{\partial{au}}{\partial{\mathbf{x}}}$	$\large a \frac{\partial{u}}{\partial{\mathbf{x}}}$	$\large a \frac{\partial{u}}{\partial{\mathbf{x}}}$
$u = u(\mathbf{x}), \; v = v(\mathbf{x})$	$\Large \frac{\partial{(u + v)}}{\partial{\mathbf{x}}}$	$\large \frac{\partial{u}}{\partial{\mathbf{x}}} + \frac{\partial{v}}{\partial{\mathbf{x}}}$	$\large \frac{\partial{u}}{\partial{\mathbf{x}}} + \frac{\partial{v}}{\partial{\mathbf{x}}}$
$u = u(\mathbf{x}), \; v = v(\mathbf{x})$	$\Large \frac{\partial{uv}}{\partial{\mathbf{x}}}$	$\large u \frac{\partial{v}}{\partial{\mathbf{x}}} + v \frac{\partial{u}}{\partial{\mathbf{x}}}$	$\large u \frac{\partial{v}}{\partial{\mathbf{x}}} + v \frac{\partial{u}}{\partial{\mathbf{x}}}$
$u = u(\mathbf{x})$	$\Large \frac{\partial{g}(u)}{\partial{\mathbf{x}}}$	$\large \frac{\partial{g}(u)}{\partial{u}} \frac{\partial{u}}{\partial{\mathbf{x}}}$	$\large \frac{\partial{g}(u)}{\partial{u}} \frac{\partial{u}}{\partial{\mathbf{x}}}$
$u = u(\mathbf{x})$	$\Large \frac{\partial{f(g(u))}}{\partial{\mathbf{x}}}$	$\large \frac{\partial{f}(g)}{\partial{g}} \frac{\partial{g}(u)}{\partial{u}} \frac{\partial{u}}{\partial{\mathbf{x}}}$	$\large \frac{\partial{f}(g)}{\partial{g}} \frac{\partial{g}(u)}{\partial{u}} \frac{\partial{u}}{\partial{\mathbf{x}}}$
$\mathbf{u} = \mathbf{u}(\mathbf{x}), \; \mathbf{v} = \mathbf{v}(\mathbf{x})$	$\Large \frac{\partial{(\mathbf{u \cdot v})}}{\partial{\mathbf{x}}}$	$\mathbf{u}^T \large \frac{\partial{\mathbf{v}}}{\partial{\mathbf{x}}} + \normalsize \mathbf{v}^T \large \frac{\partial{\mathbf{u}}}{\partial{\mathbf{x}}}$	$\large \frac{\partial{\mathbf{u}}}{\partial{\mathbf{x}}} \normalsize \mathbf{v} + \large \frac{\partial{\mathbf{v}}} {\partial{\mathbf{x}}} \normalsize \mathbf{u}$	Proof 2-1
$\mathbf{u} = \mathbf{u}(\mathbf{x}), \; \mathbf{v} = \mathbf{v}(\mathbf{x})$, $\mathbf{A}$ is not a function of $\mathbf{x}$	$\Large \frac{\partial{(\mathbf{u} \cdot \mathbf{Av})}}{\partial{\mathbf{x}}}$	$\mathbf{u}^T \mathbf{A} \large \frac{\partial{\mathbf{v}}}{\partial{\mathbf{x}}} + \normalsize \mathbf{v}^T \mathbf{A}^T \large \frac{\partial{\mathbf{u}}}{\partial{\mathbf{x}}}$	$\large \frac{\partial{\mathbf{u}}}{\partial{\mathbf{x}}} \normalsize \mathbf{A} \mathbf{v} + \large \frac{\partial{\mathbf{v}}}{\partial{\mathbf{x}}} \normalsize \mathbf{A}^T \mathbf{u}$	Proof 2-2
$\large $	$\Large \frac{\partial^2{f}}{\partial{\mathbf{x}}\partial{\mathbf{x}}^T}$	$\mathbf{H}^T$	$\mathbf{H}$	Proof 2-3
$\large \mathbf{a}$ is not a function of $\large \mathbf{x}$	$\Large \frac{\partial{(\mathbf{a \cdot x})}}{\partial{\mathbf{x}}} $	$\mathbf{a}^T $	$\mathbf{a}$	Proof 2-4
$\mathbf{A, \; a}$ are not functions of $\mathbf{x}$	$\Large \frac{\partial{(\mathbf{a}^T \mathbf{Ax})}}{\partial{\mathbf{x}}}$	$\mathbf{a}^T \mathbf{A}$	$\mathbf{A}^T \mathbf{a}$	Proof 2-5
$\mathbf{A}$ is not a function of $\mathbf{x}$	$\Large \frac{\partial({\mathbf{x}^T\mathbf{Ax}})}{\partial{\mathbf{x}}}$	$\mathbf{x}^T (\mathbf{A} + \mathbf{A}^T)$	$(\mathbf{A} + \mathbf{A}^T) \mathbf{x}$	Proof 2-6
$\mathbf{A}$ is not a function of $\mathbf{x}$	$\Large \frac{\partial^2({\mathbf{x}^T\mathbf{Ax}})}{\partial{\mathbf{x}\partial{\mathbf{x}^T}}}$	$\mathbf{A} + \mathbf{A}^T$	$\mathbf{A} + \mathbf{A}^T$	Proof 2-7
$\large $	$\Large \frac{\partial{\Vert \mathbf{x} \Vert^2}}{\partial{\mathbf{x}}}$	$2 \mathbf{x}^T$	$2 \mathbf{x}$	Proof 2-8
$\large \mathbf{a}, \mathbf{b}$ are not functions of $\large \mathbf{x}$	$\Large \frac{\partial({\mathbf{a}^T \mathbf{xx}^T \mathbf{b}})}{\partial{\mathbf{x}}}$	$\mathbf{x}^T (\mathbf{ab}^T + \mathbf{ba}^T)$	$(\mathbf{ab}^T + \mathbf{ba}^T) \mathbf{x}$	Proof 2-9
$\large \mathbf{a}$ is not a function of $\large \mathbf{x}$	$\Large \frac{\partial{\Vert \mathbf{x - a} \Vert}}{\partial{\mathbf{x}}}$	$\large \frac{(\mathbf{x - a})^T}{\Vert \mathbf{x - a} \Vert}$	$\large \frac{\mathbf{x - a}}{\Vert \mathbf{x - a} \Vert}$	Proof 2-10

向量对标量 (Vector-by-scalar)

Codition	Expression	Numerator layout	Denominator layout	Proof
$\large \mathbf{a}$ is not a function of $\large x$	$\Large \frac{\partial{\mathbf{a}}}{\partial{x}} $	$\large \mathbf{0}$	$\large \mathbf{0}$
$\large a$ is not a function of $\large x$, $\mathbf{u} = \mathbf{u}(x)$	$\Large \frac{\partial{a\mathbf{u}}}{\partial{x}}$	$\normalsize a \large \frac{\partial{\mathbf{u}}}{\partial{x}}$	$\normalsize a \large \frac{\partial{\mathbf{u}}}{\partial{x}}$
$\large \mathbf{A}$ is not a function of $\large x$. $\mathbf{u} = \mathbf{u}(x)$	$\Large \frac{\partial{\mathbf{Au}}}{\partial{x}}$	$\normalsize \mathbf{A} \large \frac{\partial{\mathbf{u}}}{\partial{x}}$	$\large \frac{\partial{\mathbf{u}}}{\partial{x}} \normalsize \mathbf{A}^T$	Proof 3-1
$\mathbf{u} = \mathbf{u}(x)$	$\Large \frac{\partial{\mathbf{u}^T}}{\partial{x}}$	$\large (\frac{\partial{\mathbf{u}}}{\partial{x}})^T$	$\large (\frac{\partial{\mathbf{u}}}{\partial{x}})^T$
$\mathbf{u} = \mathbf{u}(x), \mathbf{v} = \mathbf{v}(x)$	$\Large \frac{\partial(\mathbf{u} + \mathbf{v})}{\partial{x}}$	$\large \frac{\partial{\mathbf{u}}}{\partial{x}} + \frac{\partial{\mathbf{v}}}{\partial{x}}$	$\large \frac{\partial{\mathbf{u}}}{\partial{x}} + \frac{\partial{\mathbf{v}}}{\partial{x}}$
$\mathbf{u} = \mathbf{u}(x), \mathbf{v} = \mathbf{v}(x)$	$\Large \frac{\partial(\mathbf{u}^T \mathbf{v})}{\partial{x}}$	$\large (\frac{\partial{\mathbf{u}}}{\partial{x}})^T \normalsize \mathbf{v} + \mathbf{u}^T \large \frac{\partial{\mathbf{v}}}{\partial{x}}$	$\large \frac{\partial{\mathbf{u}}}{\partial{x}} \normalsize \mathbf{v} + \mathbf{u}^T \large (\frac{\partial{\mathbf{v}}}{\partial{x}})^T$	Proof 3-2
$\mathbf{u} = \mathbf{u}(x)$	$\Large \frac{\partial{\mathbf{g}(\mathbf{u})}}{\partial{x}}$	$\large \frac{\partial{\mathbf{g}(\mathbf{u})}}{\partial{\mathbf{u}}} \frac{\partial{\mathbf{u}}}{\partial{x}}$	$\large \frac{\partial{\mathbf{u}}}{\partial{x}} \frac{\partial{\mathbf{g}(\mathbf{u})}}{\partial{\mathbf{u}}}$	Proof 3-3
$ \mathbf{u} = \mathbf{u}(x) $	$\Large \frac{\partial{\mathbf{f}(\mathbf{g}(\mathbf{u}))}}{\partial{x}}$	$\large \frac{\partial{\mathbf{f}(\mathbf{g})}}{\partial{\mathbf{g}}} \frac{\partial{\mathbf{g}(\mathbf{u})}}{\partial{\mathbf{u}}} \frac{\partial{\mathbf{u}}}{\partial{x}}$	$\large \frac{\partial{\mathbf{u}}}{\partial{x}} \frac{\partial{\mathbf{g}(\mathbf{u})}}{\partial{\mathbf{u}}} \frac{\partial{\mathbf{f}(\mathbf{g})}}{\partial{\mathbf{g}}}$	Proof 3-4
$\mathbf{U} = \mathbf{U}(x), \mathbf{v} = \mathbf{v}(x)$	$\Large \frac{\partial{\mathbf{U}\mathbf{v}}}{\partial{x}}$	$\large \frac{\partial{\mathbf{U}}}{\partial{x}} \normalsize \mathbf{v} + \mathbf{U} \large \frac{\partial{\mathbf{v}}}{\partial{x}}$	$\large \normalsize \mathbf{v}^T \large \frac{\partial{\mathbf{U}}}{\partial{x}} + \frac{\partial{\mathbf{v}}}{\partial{x}} \normalsize \mathbf{U}^T$	Proof 3-5

推导过程

向量对向量 (Vector-by-vector)

Proof 1-1
- $\color{royalblue}{\LARGE \frac{\partial{\mathbf{a}}}{\partial{\mathbf{x}}}}$, $\; \large \mathbf{a}$ is not a function of $\large \mathbf{x}$
  \[\mathbf{a} = \begin{pmatrix} a_1 & \cdots & a_m \end{pmatrix}_{m \times 1}^T , \qquad \mathbf{x} = \begin{pmatrix} x_1 & \cdots & x_n \end{pmatrix}_{n \times 1}^T\]
  Numerator layout:
  \[\frac{\partial{\mathbf{a}}}{\partial{\mathbf{x}}} = \large \begin{pmatrix} \frac{\partial{a_1}}{\partial{x_1}} & \cdots & \frac{\partial{a_1}}{\partial{x_n}} \\ \vdots & \ddots & \vdots \\ \frac{\partial{a_m}}{\partial{x_1}} & \cdots & \frac{\partial{a_m}}{\partial{x_n}} \end{pmatrix}_{m \times n} = \begin{pmatrix} 0 & \cdots & 0 \\ \vdots & \ddots & \vdots \\ 0 & \cdots & 0 \end{pmatrix}_{m \times n} = \mathbf{0}_{m \times n}\]
  Denominator layout:
  \[\frac{\partial{\mathbf{a}}}{\partial{\mathbf{x}}} = \large \begin{pmatrix} \frac{\partial{a_1}}{\partial{x_1}} & \cdots & \frac{\partial{a_m}}{\partial{x_1}} \\ \vdots & \ddots & \vdots \\ \frac{\partial{a_1}}{\partial{x_n}} & \cdots & \frac{\partial{a_m}}{\partial{x_n}} \end{pmatrix}_{n \times m} = \begin{pmatrix} 0 & \cdots & 0 \\ \vdots & \ddots & \vdots \\ 0 & \cdots & 0 \end{pmatrix}_{n \times m} = \mathbf{0}_{n \times m}\]

关于矩阵求导（其一）

概念准备

函数与变元的种类

分子布局与分母布局

矩阵求导公式

向量对向量 (Vector-by-vector)

标量对向量 (Scalar-by-vector)

向量对标量 (Vector-by-scalar)

推导过程

向量对向量 (Vector-by-vector)

标量对向量 (Scalar-by-vector)

向量对标量 (Vector-by-scalar)

参考

Further Reading

神经网络中基于BP（反向传播）的梯度下降法

Leetcode 343 整数拆分

Windows系统下Docker的环境配置