Stable-baselines: [question] Remove bias from neural network model.

[wilsonsamarques]

Hello,

I'm using TD3 for training a MLP policy in a custom environment and I would like to know what is the appropriate way to remove the bias from the neural network model, since I would like to have a policy with just the weights. To do so, which files should I modify? Is there an easier way to simply disable the bias from the model?

Best regards,

Wilson

[Miffyli]

I am not quite sure why would you want to get rid of the biases in the network, but you can do this by creating a custom policy where you do not use any biases.

[wilsonsamarques]

Thanks for the response! But I have tried that before, however, by just specifying a custom policy by passing the parameters as keyword arguments to the model does not eliminate the bias. The resulting model still contains the bias, since I think it is defined by default when you create the tensorflow model within the toolbox. Seems to be necessary to directly modify the file where you declare the tensorflow dense layer. Am I wrong?

The bias is not interesting in my aplication since it seems to be causing a steady-state error in my model, as I posted a question here in the forum some time ago. https://github.com/hill-a/stable-baselines/issues/963
With the bias, the neural network may send a zero torque even if there are still error (satellite pointing to the wrong direction). A PD law would never do such a thing. So it does not seems to be suitable to use bias in the model in the context of a regulator control problem.

[Miffyli]

Ah yes. If you want the hackiest solution then you need to stab functions in this file to remove the bias terms (e.g. modify this function call and update this part to remove bias, and same tricks for other layers you need). A cleaner solution would be to create a custom policy network but use modified tools to create the convolution/linear/etc layers without the bias term.

[wilsonsamarques]

Great! I'll try that out. Thanks!

[araffin]

I would also recommend you to give Stable-Baselines3 a try ;)
It should be much easier to hack and SAC/TD3 are also included.

[wilsonsamarques]

Yes, I saw that you decided to migrate to PyTorch. They changed so much with Tensorflow 2.0... I think it is indeed better to invest time learning to use PyTorch. I will give it a try! Thanks for the tip! May I have both versions of the toolbox installed in the same Anaconda Environment?

[Miffyli]

@wilsonsamarques Yes, you can have both versions of stable-baselines installed in same environment (they have separate names), along with Tensorflow and PyTorch.

You may close this issue if there are no further questions.

[wilsonsamarques]

Ah yes. If you want the hackiest solution then you need to stab functions in this file to remove the bias terms (e.g. modify this function call and update this part to remove bias, and same tricks for other layers you need). A cleaner solution would be to create a custom policy network but use modified tools to create the convolution/linear/etc layers without the bias term.

Hello,

I made the modifications you recommended to remove the bias. I simply passed the parameter use_bias=False in the respective tensorflow dense layers declaration. However, I'm not sure if it actually worked. Since I don't fully comprehend the parameters of the trained model yet (since there are many) when I check with the functions get_parameters() and get_parameter_list. I will send a print of the output of these functions. Can you help me check if the bias was successfully removed?

I defined the policy network as a MLP with 2 hidden layers. And my model receives 3 inputs and returns 1 output.

This is the output of the get_parameter_list() function:

And these are the outputs of the get_parameters() function:

Best regards,
Wilson

[Miffyli]

Seems correct to me. "kernel" refers to weights between nodes, and biases should be "bias".

[wilsonsamarques]

That is good to know! Thanks a lot for the feedback! Unfortunately, removing the bias still didn't solve my problem, hence it may be some problem with my custom environment. However, I had to make this test to be certain. But I will close this issue now.

All the best,
Wilson