(No title) (answer)

What is the derivative of negative log MLE (MLE used as a negative cost) when the variables are passed through softmax activations?

The derivative of the cost function is needed for back-propagation.

When the output layer uses a MLE/cross-entropy cost and softmax activation, the derivative of the cost function combined with the activation function simplifies to a very simple expression. This note covers this computation.

Consider the last layer of a neural network shown below:

For the given training example $x_{1}$ , the correct category is 2 (out of all categories 0,1 and 2). Only the output corresponding to category 2 is used in the calculation of the loss contributed by the example. The loss is given by:

L o s s_{e x_{1}} = - l n (p_{3}) = - l n (\frac{e^{f_{3}}}{e^{f_{1}} + e^{f_{2}} + e^{f_{3}}})

While only the probability for the correct category is used to calculate the loss for the example, the other outputs, $p_{1}$ and $p_{2}$ still affect the loss, as $p_{3}$ depends on these values (all sum to 1).

The partial derivatives will be:

\begin{aligned} \frac{\partial L}{\partial f_{1}} & = p_{1} \\ \frac{\partial L}{\partial f_{2}} & = p_{2} \\ \frac{\partial L}{\partial f_{3}} & = - (1 - p_{3}) = p_{3} - 1 \end{aligned}

So if we have output probabilities [0.1, 0.3, 0.6], then the derivative of the loss with respect to last layer's output before applying the softmax activation will be [0.1, 0.3, -0.4].

Derivations

Formulating the derivative of loss wrt different variables is simplified if the log and exponential are canceled at the beginning:

\begin{aligned} L o s s_{e x_{1}} = - l n (p_{3}) & = - l n (\frac{e^{f_{3}}}{e^{f_{1}} + e^{f_{2}} + e^{f_{3}}}) \\ = l n (\frac{e^{f_{1}} + e^{f_{2}} + e^{f_{3}}}{e^{f_{3}}}) \\ = l n (e^{f_{1}} + e^{f_{2}} + e^{f_{3}}) - l n (e^{f_{3}}) \\ = l n (e^{f_{1}} + e^{f_{2}} + e^{f_{3}}) - f_{3} \end{aligned}

From here, the partial derivates are obvious. For example:

\begin{aligned} \frac{\partial L}{\partial f_{1}} & = \frac{\partial}{\partial f_{1}} (l n (e^{f_{1}} + e^{f_{2}} + e^{f_{3}}) - f_{3}) \\ = \frac{\partial}{\partial x} (l n (x)) \frac{\partial}{\partial f_{1}} (e^{f_{1}} + e^{f_{2}} + e^{f_{3}}) \\ = \frac{1}{x} e^{f_{1}} \\ = \frac{e^{f_{1}}}{e^{f_{1}} + e^{f_{2}} + e^{f_{3}}} \\ = p_{1} \end{aligned}

The below image is part of a separate card, but I'm including it here, as it is quite relevant.

Old image:

Some old stuff:

Given the loss (lets just call it L, with the example 1 being implicit), we wish to find: $\frac{\partial L}{\partial f_1} \text{, } \frac{\partial L}{\partial f_2} \text{, } \frac{\partial L}{\partial f_3}$ . First consider $f_1$ :

\begin{aligned} \frac{\partial L}{\partial f_{1}} = \frac{\partial}{\partial f_{1}} (- l n (p_{3})) & = (\frac{\partial}{\partial x} (- l n (x)) (\frac{\partial}{\partial y} (\frac{e^{f_{3}}}{y + e^{f_{2}} + e^{f_{3}}})) (\frac{\partial}{\partial f_{1}} (e^{f_{1}})) \\ where x = \frac{e^{f_{3}}}{y + e^{f_{2}} + e^{f_{3}}}, and y = e^{f_{1}} \\ = (- \frac{1}{x}) (\frac{- e^{f_{3}}}{(y + e^{f_{2}} + e^{f_{3}})^{2}}) (e^{f_{1}}) \\ = (- \frac{e^{f_{1}} + e^{f_{2}} + e^{f_{3}}}{e^{f_{3}}}) (\frac{- e^{f_{3}}}{(e^{f_{1}} + e^{f_{2}} + e^{f_{3}})^{2}}) (e^{f_{1}}) \\ = \frac{e^{f_{1}}}{e^{f_{1}} + e^{f_{2}} + e^{f_{3}}} \\ = p_{1} \end{aligned}

Similarly, $\frac{\partial L}{\partial f_2} = p_2$ . $\frac{\partial L}{\partial f_3}$ is calculated as follows:

\begin{aligned} \frac{\partial L}{\partial f_{3}} = \frac{\partial}{\partial f_{3}} (- l n (p_{3})) & = (\frac{\partial}{\partial x} (- l n (x)) (\frac{\partial}{\partial y} (\frac{e^{f_{3}}}{e^{f_{1}} + e^{f_{2}} + y})) (\frac{\partial}{\partial f_{3}} (e^{f_{3}})) \\ where x = \frac{e^{f_{3}}}{e^{f_{1}} + e^{f_{2}} + y}, and y = e^{f_{3}} \\ = (- \frac{1}{x}) (\frac{1}{e^{f_{1}} + e^{f_{2}} + y} - \frac{e^{f_{3}}}{e^{f_{1}} + e^{f_{2}} + y}) (e^{f_{3}}) \\ = (- \frac{e^{f_{1}} + e^{f_{2}} + e^{f_{3}}}{e^{f_{3}}}) (\frac{1}{e^{f_{1}} + e^{f_{2}} + e^{f_{3}}} - \frac{e^{f_{3}}}{e^{f_{1}} + e^{f_{2}} + e^{f_{3}}}) (e^{f_{3}}) \\ = - (1 - \frac{e^{f_{3}}}{e^{f_{1}} + e^{f_{2}} + e^{f_{3}}}) \\ = p_{3} - 1 \end{aligned}

19.03.2019