Probability: With TF 2.0 preview, "old-style" variational layers error with "Inputs to eager execution function cannot be Keras symbolic tensors"

[skeydan]

Hi,

I have unit tests for Convolution1DReparameterization , Convolution1DReparameterization etc. that basically look like this

x = tf.ones(shape = [150,1])
y = tf.ones(shape = [150])

model = tf.keras.Sequential(
  [tfpl.DenseReparameterization(
      units = 512,
      activation = "relu"),
   tfpl.DenseReparameterization(
      units = 1)])

model.compile(optimizer = 'adam', loss = "mse")
model.fit(x, y, steps_per_epoch = 1)

These run fine with TF 1, but with TF 2 preview I get

_SymbolicException: Inputs to eager execution function cannot be Keras symbolic tensors, but found [<tf.Tensor 'sequential/dense_reparameterization/divergence_kernel:0' shape=() dtype=float32>, <tf.Tensor 'sequential/dense_reparameterization_1/divergence_kernel:0' shape=() dtype=float32>]

Would you happen to know what is the cause here?
Are these "old-style" variational layers supposed to run with TF 2?

Many thanks in advance!!

[jburnim]

These layers should be working in TF2.

It looks like there is an incompatibility with these layers and a recent change in Keras. We will investigate.

If you add experimental_run_tf_function=False as an argument to model.compile, do the errors go away?

[skeydan]

Thanks, and

If you add experimental_run_tf_function=False as an argument to model.compile, do the errors go away?

yes they do!

[vgoklani]

any reason why we have to use "experimental_run_tf_function=False". I need to use this too.

[skeydan]

In the TF 2.0-rc0 release notes I saw this

Switched Keras fit/evaluate/predict execution to use only a single unified path by default unless eager execution has been explicitly disabled, regardless of input type. This unified path places an eager-friendly training step inside of a tf.function. With this 1. All input types are converted to Dataset. 2. The path assumes there is always a distribution strategy. when distribution strategy is not specified the path uses a no-op distribution strategy. 3. The training step is wrapped in tf.function unless run_eagerly=True is set in compile. The single path execution code does not yet support all use cases. We fallback to the existing v1 execution paths if your model contains the following: 1. sample_weight_mode in compile 2. weighted_metrics in compile 3. v1 optimizer 4. target tensors in compile. If you are experiencing any issues because of this change, please inform us (file an issue) about your use case and you can unblock yourself by setting experimental_run_tf_function=False in compile meanwhile. We have seen couple of use cases where the model usage pattern is not as expected and would not work with this change. 1. output tensors of one layer is used in the constructor of another. 2. symbolic tensors outside the scope of the model are used in custom loss functions. The flag can be disabled for these cases and ideally the usage pattern will need to be fixed.

So guessing (haven't checked) it might be "output tensors of one layer is used in the constructor of another."

However @jburnim as far as I can see these layers don't really work with TF 2, do they (assuming eager mode)?
I tried Keras-style fit which seemed to run fine syntactically, but didn't accumulate the losses (and furthermore, doesn't allow for scaling the KL), as well as GradientTape which gave an error (for all this see mails to the list: https://groups.google.com/a/tensorflow.org/forum/#!topic/tfprobability/axQpJqs0oHc ).

Is there a way to use these layers in TF 2, and if so, could you please indicate it? Otherwise, are there plans to create V2 versions of these layers?

Thanks!

[tomtang110]

experimental_run_tf_function=False

but If I don't use model.compile, how do I do?
`class PolicyEstimator():

def __init__(self, lr=1e-2, name='policy_estimator'):
    self.state = tf.keras.Input(shape=(400,), name='state', dtype=tf.float32)
    self.mu = layers.Dense(1, activation=None,kernel_initializer='random_normal')(self.state)
    self.mu = tf.squeeze(self.mu)
    self.sigma = layers.Dense(1, activation=None,kernel_initializer='random_normal')(self.state)
    self.sigma = tf.squeeze(self.sigma)
    self.sigma = tf.keras.activations.softplus(self.sigma) + 1e-5
    self.normal_dist = tfp.distributions.Normal(self.mu, self.sigma)
    self.action = self.normal_dist._sample_n(1)
    self.action = tf.clip_by_value(self.action, env.action_space.low[0], env.action_space.high[0])
    self.model = tf.keras.Model(inputs=self.state, outputs=self.action)
    self.optimizer = tf.keras.optimizers.Adam(lr)

def predict(self, state):
    return self.model(state)

def update(self, state, target, action):
    with tf.GradientTape() as tape:
        predictions = self.model(state)[0]
        loss = -self.normal_dist.log_prob(predictions) # this would produce a non-eager tensor
        loss -= 1e-1 * self.normal_dist.entropy()
        print('policy: ', loss)
        grads = tape.gradient(loss, self.model.trainable_variables)
        self.optimizer.apply_gradients(zip(grads, self.model.trainable_variables))
    return loss`

As you can be seen, the loss in update function is produced by self.normal_dist.log_prob, which would produce the error-"Inputs to eager execution function cannot be Keras symbolic tensors,but found []".
I did not use compile function so how can I solve the problem?

[Crispy13]

These layers should be working in TF2.

It looks like there is an incompatibility with these layers and a recent change in Keras. We will investigate.

If you add experimental_run_tf_function=False as an argument to model.compile, do the errors go away?

I added it as an argument to model.compile, but didn't work.

[felixvelariusbos]

Just an added update here for version: This issue is still in TensorFlow 2.1, but the provided workaround (experimental_run_tf_function=False) still works

[William-YanHua]

These layers should be working in TF2.

It looks like there is an incompatibility with these layers and a recent change in Keras. We will investigate.

If you add experimental_run_tf_function=False as an argument to model.compile, do the errors go away?

I add this argument to 'model.compile', it can be compiled successfully, but when it execution with ModelCheckpoint and EarlyStopping, it will cause the same problem while saving the checkpoint.

[moein-bit]

These layers should be working in TF2.

It looks like there is an incompatibility with these layers and a recent change in Keras. We will investigate.

If you add experimental_run_tf_function=False as an argument to model.compile, do the errors go away?

thank you so much. I had a similar problem and it's gone now

[Zumbalamambo]

It does not work anymore :(

[lastmansleeping]

Facing the same issue with TF2.2.0. Looking at this #35138, I see that there was a major API change that was made. Can you recommend how we can bypass this in the latest stable release?

[Zumbalamambo]

@lastmansleeping simply migrate to pytorch. Took this decision after wasting weeks to fix.

[krisztinasinkovics]

Just use this line. This will change the input type to EagerTensor.
tf.config.experimental_run_functions_eagerly(True)

[arunjose696]

Thanks @Krisztina-Sinkovics it works... Could you provide an explanation why this is required.

[Vivek-1116]

Just use this line. This will change the input type to EagerTensor.
tf.config.experimental_run_functions_eagerly(True)

Getting this error after using this line of code...can anyone clarify?

[CodTan]

I had the same issue, and was earlier working on TF 2.2, and then on TF 2.3.0
I tried following the above solutions, but none of them worked for me.
What worked for me though was
`` import tensorflow as tf tf.compat.v1.disable_eager_execution() ```` Note: The model instance and call to.fit()` must both be in Graph mode, otherwise it will result in incompatibility. This means, run the above code snippet first and then instantiate your model.

Found the solution here: https://datascience.stackexchange.com/a/80599

[Zumbalamambo]

@Vivek-1116 I just switched to Pytorch since the arrival of Tf2. Tensorflow v2 is full of bugs. I tried all these approaches that are mentioned here and I also tried tf-upgrade package but nothing really worked. Pytorch is backward compatible, unlike TensorFlow. So, It's better to use Pytorch than Tensorflow for production. On top of that, it's quite simple as well.

[qoolcezar]

I have the same issue with a actor-critic setup but I managed a workaround.

class Agent(object):
    def __init__(self, alpha, beta, gamma=0.99, n_actions=5, layer1_size=16, layer2_size=32, input_dims=4):
        self.gamma = gamma
        self.alpha = alpha
        self.beta = beta
        self.input_dims = input_dims
        self.fc1_dims = layer1_size
        self.fc2_dims = layer2_size
        self.n_actions = n_actions
        self.action_space = [i for i in range(self.n_actions)]
        self.delta_in = tf.Variable([[1.]])

        try:
            self.actor = tf.keras.models.load_model('model_saves/actor_model.h5', custom_objects={'custom_loss': self.delta_loss()})
            self.critic = tf.keras.models.load_model('model_saves/critic_model.h5')
            self.policy = tf.keras.models.load_model('model_saves/policy_model.h5', compile=False)
            print('Agent -> model loaded')
        except:
            self.actor, self.critic, self.policy = self.build_actor_critic_network()

    def save_model(self):
        self.actor.save('model_saves/actor_model.h5') 
        self.critic.save('model_saves/critic_model.h5') 
        self.policy.save('model_saves/policy_model.h5') 

    def delta_loss(self):
        def custom_loss(y_true, y_pred):
            out = K.clip(y_pred, 1e-8, 1-1e-8)
            log_lik = y_true*K.log(out)
            return K.sum(-log_lik*self.delta_in)
        custom_loss.__name__='custom_loss'
        return custom_loss

    def build_actor_critic_network(self):
        inp = Input(shape=(self.input_dims, ))
        dense1 = Dense(self.fc1_dims, activation='relu')(inp)
        dense2 = Dense(self.fc2_dims, activation='relu')(dense1)
        probs = Dense(self.n_actions, activation='softmax')(dense2)
        values = Dense(1, activation='linear')(dense2)

        actor = Model(inputs=[inp], outputs=[probs], name='actor')
        actor.compile(optimizer=Adam(lr=self.alpha), loss=self.delta_loss(), metrics=['accuracy'])

        critic = Model(inputs=[inp], outputs=[values], name='critic')
        critic.compile(optimizer=Adam(lr=self.beta), loss='mean_squared_error')

        policy = Model(inputs=[inp], outputs=[probs])

        return actor, critic, policy

    def choose_action(self, observation):
        state = np.array(observation)
        state = state[np.newaxis, :]
        probabilities = self.policy.predict(state)[0]
        action = np.random.choice(self.action_space, p=probabilities)

        return action

    def learn(self, state, action, reward, state_, done):
        state = np.array(state, dtype=np.float16)
        state_ = np.array(state_, dtype=np.float16)
        reward = np.array(reward, dtype=np.float16)
        state = state[np.newaxis, :]
        state_ = state_[np.newaxis, :]

        critic_value_ = self.critic.predict(state_)
        critic_value = self.critic.predict(state)

        target = reward + self.gamma*critic_value_*(1-int(done))
        delta = target - critic_value
        self.delta_in.assign(delta)

        actions = np.zeros([1, self.n_actions], dtype=np.int8)
        actions[np.arange(1), action] = 1.0

        if False:
            self.actor.fit(state, actions, verbose=1)
            self.critic.fit(state, target, verbose=1)
        else:
            actor_thread = Thread(target=self.actor.fit, kwargs={'x':state, 'y':actions, 'verbose':1})
            critic_thread = Thread(target=self.critic.fit, kwargs={'x':state, 'y':target, 'verbose':1})
            actor_thread.start()
            critic_thread.start()
            actor_thread.join()
            critic_thread.join()

[tao2020]

I have the same issue with a actor-critic setup but I managed a workaround.

class Agent(object):
    def __init__(self, alpha, beta, gamma=0.99, n_actions=5, layer1_size=16, layer2_size=32, input_dims=4):
        self.gamma = gamma
        self.alpha = alpha
        self.beta = beta
        self.input_dims = input_dims
        self.fc1_dims = layer1_size
        self.fc2_dims = layer2_size
        self.n_actions = n_actions
        self.action_space = [i for i in range(self.n_actions)]
        self.delta_in = tf.Variable([[1.]])

        try:
            self.actor = tf.keras.models.load_model('model_saves/actor_model.h5', custom_objects={'custom_loss': self.delta_loss()})
            self.critic = tf.keras.models.load_model('model_saves/critic_model.h5')
            self.policy = tf.keras.models.load_model('model_saves/policy_model.h5', compile=False)
            print('Agent -> model loaded')
        except:
            self.actor, self.critic, self.policy = self.build_actor_critic_network()

    def save_model(self):
        self.actor.save('model_saves/actor_model.h5') 
        self.critic.save('model_saves/critic_model.h5') 
        self.policy.save('model_saves/policy_model.h5') 

    def delta_loss(self):
        def custom_loss(y_true, y_pred):
            out = K.clip(y_pred, 1e-8, 1-1e-8)
            log_lik = y_true*K.log(out)
            return K.sum(-log_lik*self.delta_in)
        custom_loss.__name__='custom_loss'
        return custom_loss

    def build_actor_critic_network(self):
        inp = Input(shape=(self.input_dims, ))
        dense1 = Dense(self.fc1_dims, activation='relu')(inp)
        dense2 = Dense(self.fc2_dims, activation='relu')(dense1)
        probs = Dense(self.n_actions, activation='softmax')(dense2)
        values = Dense(1, activation='linear')(dense2)

        actor = Model(inputs=[inp], outputs=[probs], name='actor')
        actor.compile(optimizer=Adam(lr=self.alpha), loss=self.delta_loss(), metrics=['accuracy'])

        critic = Model(inputs=[inp], outputs=[values], name='critic')
        critic.compile(optimizer=Adam(lr=self.beta), loss='mean_squared_error')

        policy = Model(inputs=[inp], outputs=[probs])

        return actor, critic, policy

    def choose_action(self, observation):
        state = np.array(observation)
        state = state[np.newaxis, :]
        probabilities = self.policy.predict(state)[0]
        action = np.random.choice(self.action_space, p=probabilities)

        return action

    def learn(self, state, action, reward, state_, done):
        state = np.array(state, dtype=np.float16)
        state_ = np.array(state_, dtype=np.float16)
        reward = np.array(reward, dtype=np.float16)
        state = state[np.newaxis, :]
        state_ = state_[np.newaxis, :]

        critic_value_ = self.critic.predict(state_)
        critic_value = self.critic.predict(state)

        target = reward + self.gamma*critic_value_*(1-int(done))
        delta = target - critic_value
        self.delta_in.assign(delta)

        actions = np.zeros([1, self.n_actions], dtype=np.int8)
        actions[np.arange(1), action] = 1.0

        if False:
            self.actor.fit(state, actions, verbose=1)
            self.critic.fit(state, target, verbose=1)
        else:
            actor_thread = Thread(target=self.actor.fit, kwargs={'x':state, 'y':actions, 'verbose':1})
            critic_thread = Thread(target=self.critic.fit, kwargs={'x':state, 'y':target, 'verbose':1})
            actor_thread.start()
            critic_thread.start()
            actor_thread.join()
            critic_thread.join()

please explain what you changed and why, so that people don't need to parse your whole code.

[mohamedmlh]

I had the same issue, and was earlier working on TF 2.2, and then on TF 2.3.0
I tried following the above solutions, but none of them worked for me.
What worked for me though was

import tensorflow as tf
tf.compat.v1.disable_eager_execution()

Note: The model instance and call to .fit() must both be in Graph mode, otherwise it will result in incompatibility. This means, run the above code snippet first and then instantiate your model.

Found the solution here: https://datascience.stackexchange.com/a/80599

it worked for me thanks