Handson-ml: Chapter 14 - Why was no activation function used in the fully connected layer attached after the final state for sequence classification?

[kusur]

I was going through the code for building sequence classifier using MNIST dataset. While the entire architecture made sense to me, what continues to puzzle me is why was there no activation used in the fully connected layer, especially when there is not further layer that contains softmax activation function. The code is as follows -

n_steps = 28
n_inputs = 28
n_neurons = 150
n_outputs = 10

learning_rate = 0.001

X = tf.placeholder(tf.float32, [None, n_steps, n_inputs])
y = tf.placeholder(tf.int32, [None])

basic_cell = tf.nn.rnn_cell.BasicRNNCell(num_units=n_neurons)
outputs, states = tf.nn.dynamic_rnn(basic_cell, X, dtype=tf.float32)

logits = tf.layers.dense(states, n_outputs)  ## issue that is unclear to me
xentropy = tf.nn.sparse_softmax_cross_entropy_with_logits(labels=y,
                                                          logits=logits)
loss = tf.reduce_mean(xentropy)
optimizer = tf.train.AdamOptimizer(learning_rate=learning_rate)
training_op = optimizer.minimize(loss)
correct = tf.nn.in_top_k(logits, y, 1)
accuracy = tf.reduce_mean(tf.cast(correct, tf.float32))

init = tf.global_variables_initializer()

The output generated by logits is used in calculating the cross entropy. Why was this choice made? Could you please shed some light on it.

Thanks

[ageron]

Hi @kusur,
Great question! When you want to output class probabilities, in general the loss function to use is the Cross-Entropy. In pseudo-code:
logits = some_complex_function(inputs)
y_probas = softmax(logits)
loss = cross_entropy(y_probas)

However, computing the softmax involves computing the exponential of every logit. And computing the cross_entropy involves computing the log of every probability. So it's more efficient and numerically stable to simplify the equations and compute the cross-entropy directly based on the logits.
This is why we use the sparse_softmax_cross_entropy_with_logits() loss function: it computes the cross-entropy loss based on the logits instead of the output probabilities.
The logits do not require any activation function.
In this code example, since we only care about which class has the highest output probability (but we don't care about what that probability is), we don't even need to compute the output probability, we just look at the class with the highest logit. This is why we don't even call the softmax function anywhere.
Hope this helps!

[kusur]

That's interesting! I just have a question though -

As far as I know, RNN uses tanh as an activation function. Now, tanh has a range of [-1,1]. So, this means that an activation can be negative as well. If there is no activation applied in the subsequent dense layer, a negative value could come out as a negative value. Won't this be a potential problem while calculating cross entropy as log of a negative value is not defined?

[ageron]

Hi @kusur ,
Interesting question! Okay, I oversimplified things a bit. Let's go more into the math. The cross-entropy is:

H = -sum(p(c) * log(q(c)))
The sum is over all classes c (as all the sums below). And p(c) is the target probability for class c, and q(c) is the estimated probability for class c.

Now using the equation for softmax, we have:
q(k)=exp(g(k))/sum(exp(g(c)))
where g(c) is the logit for class c.

If G=max(g(c)) over all classes c, we can rewrite this equation like this:
q(k)=exp(g(k))/sum(exp(g(c) - G)) * exp(G)/exp(G)
So:
q(k)=exp(g(k))/exp(G) * exp(G)/sum(exp(g(c) - G))
And since exp(a)/exp(b) = exp(a-b), we get:
q(k)=exp(g(k)-G) / sum(exp(g(c) - G))
The benefit is that now we don't have potentially huge exponentials which could wreak computations.

Now let's look at the log(q(c)) component of the cross-entropy:
log(q(k)) = log[exp(g(k)-G) / sum(exp(g(c) - G))]
and since log(a/b)=log(a)-log(b):
log(q(k)) = log(exp(g(k)-G)) - log(sum(exp(g(c)-G)))
log(q(k)) = g(k) - G - log(sum(exp(g(c) - G)))

So TensorFlow will compute:

G = max(g(c))
A = log(sum(exp(g(c) - G)))
H = -sum(p(c) * [g(c) - G - A])

Not 100% sure (I'd have to check the source code), but I think that's about it. As you can see, we never compute the log of a negative value. There's a nice post about this btw.
Hope this helps!

[kusur]

This is such a neat trick!. Thanks a ton for your help. Closing this issue :)