Handson-ml: chapter 11 Question 8 restore_model parameter function ?

[nitml]

in chapter 11 Question 8 the code written to restore model parameter to value of best parameters in early stop is :

    def _restore_model_params(self, model_params):
        gvar_names = list(model_params.keys())
        assign_ops = {gvar_name: self._graph.get_operation_by_name(gvar_name + "/Assign")
                      for gvar_name in gvar_names}
        init_values = {gvar_name: assign_op.inputs[1] for gvar_name, assign_op in assign_ops.items()}
        feed_dict = {init_values[gvar_name]: model_params[gvar_name] for gvar_name in gvar_names}
        self._session.run(assign_ops, feed_dict=feed_dict)

I'm not able to understand this code properly .Can you please explain assign_ops and init_values code ?

[ageron]

Hi @nitml,

Good question. When you create a variable, TensorFlow automatically creates several operations and tensors, including an assignment operation to initialize the variable to whatever initialization value you chose for that variable.

• The first line of the function gets the list of names of variables whose values we want to restore.
• The next line gets a reference to all the assignment operations for these variables. Indeed, if a variable is named "V", then the assignment operation is called "V/Assign".
• The third line gets a reference to the tensors that represent the initialization values for these variables. Indeed, an assignment operation has two inputs: (1) a reference to the variable, (2) the value to assign. Here, I am getting each assignment operation's second input (op.inputs[1]).
• Then I prepare a feed_dict to tell TensorFlow to use the given model_params values instead of the initialization values for each variable.
• Then I run all the assignment operations, feeding the values to use instead of the default initialization values.

Here's a simpler example, to help you understand what's going on:

>>> import tensorflow as tf
>>> V = tf.Variable(42, name="V")
>>> graph = tf.get_default_graph()
>>> sess = tf.InteractiveSession()
>>> V.initializer.run()
>>> V.eval()
42
>>> assign_init_value_op = graph.get_operation_by_name("V/Assign")
>>> init_value = assign_init_value_op.inputs[1]
>>> init_value.eval()
42
>>> sess.run(assign_init_value_op, feed_dict={init_value: 100})
>>> V.eval()
100
>>> init_value.eval()
42

Hope this helps,
Aurélien

[nitml]

Great Explanation, i have another doubt in this Question 8 to use RandomizedSearchCV we created Sklearn learn compatible class DNNClassifier and then used following code :

from sklearn.model_selection import RandomizedSearchCV

def leaky_relu(alpha=0.01):
    def parametrized_leaky_relu(z, name=None):
        return tf.maximum(alpha * z, z, name=name)
    return parametrized_leaky_relu

param_distribs = {
    "n_neurons": [10, 30, 50, 70, 90, 100, 120, 140, 160],
    "batch_size": [10, 50, 100, 500],
    "learning_rate": [0.01, 0.02, 0.05, 0.1],
    "activation": [tf.nn.relu, tf.nn.elu, leaky_relu(alpha=0.01), leaky_relu(alpha=0.1)],
    # you could also try exploring different numbers of hidden layers, different optimizers, etc.
    #"n_hidden_layers": [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10],
    #"optimizer_class": [tf.train.AdamOptimizer, partial(tf.train.MomentumOptimizer, momentum=0.95)],
}

rnd_search = RandomizedSearchCV(DNNClassifier(random_state=42), param_distribs, n_iter=50,
                                random_state=42, verbose=2)
rnd_search.fit(X_train1, y_train1, X_valid=X_valid1, y_valid=y_valid1, n_epochs=1000)

My question is why have we defined leaked_elu() function such that it has another function parametrized_leaky_relu() inside it and not simply :

def leaky_relu(z,alpha=0.01,name=None):
      return tf.maximum(alpha * z, z, name=name)

[ageron]

Hi @nitml ,

With your leaky_relu() function, how would you define the param_distribs dict? For example, the following would NOT work, because the leaky_relu() function would complain that the argument z is missing:

# THIS WILL NOT WORK
param_distribs = {
    ...
    "activation": [tf.nn.relu, tf.nn.elu, leaky_relu(alpha=0.01), leaky_relu(alpha=0.1)],
    ...
}

Of course you could try to pass some inputs z, but which ones? We don't have that data yet, as it will only be computed during training, and it will be different for each training batch.

However, you could do this:

# THIS SHOULD WORK
param_distribs = {
    ...
    "activation": [tf.nn.relu, tf.nn.elu, lambda z: leaky_relu(z, alpha=0.01), lambda z: leaky_relu(alpha=0.1)],
    ...
}

As you can see, the activation function needs to be, well, a function, not the result of calling the leaky_relu() function. My function creates a parametrized_leaky_relu() function that has the desired alpha hyperparameter already set to the desired value. So TensorFlow will just give it the input data, and it will compute the right thing. Your function directly returns a result, so it must be wrapped into a function, for example using a lambda.

I hope this is clear,
Aurélien

[nitml]

Thanks for your time and explanation, but things are not properly clear,
suppose I want to use leaky_relu() having parametrized_leaky_relu() function within it as activation function then my code will be:

Z = tf.layers.dense(X, 20, name="hidden_layer2")
hidden2 = leaky_relu(Z)

then will not it be a error as leaky_relu() function does not have parameter Z as it is parameter of parametrized_leaky_relu() function and not leaky_relu().

In question 8 we used _dnn() method in fit() method to make layers :

def _dnn(self, inputs):
        for layer in range(self.n_hidden_layers):
            if self.dropout_rate:
                inputs=tf.layers.dropout(inputs,self.dropout_rate,training=self.training)
            inputs=tf.layers.dense(inputs,self.n_neurons,
                                   kernel_initializer=self.initializer,
                                   name="hidden%d" % (layer + 1))
            if self.batch_norm_momentum:
                inputs=tf.layers.batch_normalization(
                    inputs,
                    momentum=self.batch_norm_momentum,
                    training=self.training)
            inputs = self.activation(inputs, name="hidden%d_out" % (layer + 1))
        return inputs

so when we use RandomizedSearchCV(DNNClassifier) with activation function as leaky_relu() then second last line inputs = self.activation(inputs, name="hidden%d_out" % (layer + 1)) should raise error as inputs is parameter of parametrized_leaky_relu() function and not leaky_relu().

i hope my question is clear ?
Thanks for all your Time and support !!

[ageron]

Hi @nitml ,

Sorry I wasn't clear. As you can see on the second to last line in the _dnn() method, the self.activation attribute needs to be a function that takes some inputs as its first argument, and also takes a name argument.

For example, I could use this function if I want alpha=0.01:

def leaky_relu_001(z, name=None):
      return tf.maximum(0.01 * z, z, name=name)

Then I could just create a DNNClassifier like this:

dnn_clf = DNNClassifier(activation=leaky_relu_001, ...)

If I want to use alpha=0.1 instead, then I can create another function:

def leaky_relu_01(z, name=None):
      return tf.maximum(0.1 * z, z, name=name)

I could then create a param_distribs dict like this:

param_distribs = {
    ...
    "activation": [tf.nn.relu, tf.nn.elu, leaky_relu_001, leaky_relu_01],
    ...
}

And then create a RandomizedSearchCV using this param_distrib, and everything should work fine. Now instead of manually copy/pasting the code to write a new function every time I want a different alpha value, I could write a create_leaky_relu_function() function, that creates a leaky_relu() function for me, with the desired alpha value:

def create_leaky_relu_function(alpha):
    def leaky_relu(z, name=None):
          return tf.maximum(alpha * z, z, name=name)
    return leaky_relu

Now if I want a new function for alpha=0.2, it is much easier to create:

leaky_relu_02 = create_leaky_relu_function(0.2)

Now this leaky_relu_02() function is a function that takes the inputs z and an optional name. I can use it when creating a DNNClassifier(activation=leaky_relu_02), and I can use it in param_distribs={"activation": [leaky_relu_02, ...], ...}.

The main difficulty is to understand that the create_leaky_relu_function() returns a new function with its own alpha value that will never change once the function is created. Perhaps this simple example may help:

>>> def create_multiplier(factor):
...   def mul(x):
...     return factor * x
...   return mul
...
>>> mul2 = create_multiplier(2)
>>> mul3 = create_multiplier(3)
>>> mul2(10)
20
>>> mul3(10)
30

First I create a function that can be used to create functions. Then I use it to create two functions mul2() and mul3(). The first one has factor=2, so it multiplies its argument by 2, while the second has factor=3 so it multiplies its argument by 3.

Now let's look at your function:

def your_leaky_relu(z, alpha=0.01, name=None):
      return tf.maximum(alpha * z, z, name=name)

It can indeed be used as you showed:

Z = tf.layers.dense(X, 20, name="hidden_layer2")
hidden2 = your_leaky_relu(Z)

You could even set alpha to any value you want:

Z = tf.layers.dense(X, 20, name="hidden_layer2")
hidden2 = your_leaky_relu(Z, alpha=0.2)

However, suppose you want to pass it to the DNNClassifier constructor, and you want alpha=0.2, how can you do that? If you write the following code, then alpha is not set, so it will default to 0.01:

dnn_clf = DNNClassifier(activation=your_leaky_relu, ...)

But if you write the following code, then you will get an error, because activation is supposed to be a function, not a tensor (which is what your function returns):

dnn_clf = DNNClassifier(activation=your_leaky_relu(alpha=0.2), ...) # ERROR

The error will actually not happen right away, but only when self.activation is used (when you fit the DNNClassifier).
So, as you can see, the problem is that there is no way to set the alpha argument in your function when you pass it to the DNNClassifier. This is why, if we want to try 3 different alpha values, we need 3 different functions, and this is why it is useful to have a function that creates them.

Hope this helps,
Aurélien

[nitml]

Great Explanation,
Thanks for your time and support