Tflearn: InvalidArgumentError: Nan in summary histogram for: Adam/HistogramSummary error when using tensorboard_verbose>0

[llealgt]

Hi, im having an issue when i change tensorboard_verbose to a value bigger than 0, im getting the error:
InvalidArgumentError: Nan in summary histogram for: Adam/HistogramSummary

If i use tensorboard_verbose = 0 everything works fine,any tips?, heres my code:

``````
network = input_data(shape=[None,image_rows,image_cols,1],name='input');
network = conv_2d(network, 32, 3, activation='relu',padding='same');
network = conv_2d(network, 32, 3, activation='relu',padding='same');
network = max_pool_2d(network,2);#probar solo con 2 en vez de [2,2]
network = conv_2d(network, 32, 3, activation='relu',padding='same');
network = max_pool_2d(network,2);
network = fully_connected(network,512,activation ="relu");
network = dropout(network, 0.5);
network = fully_connected(network, 1, activation='sigmoid')

network = regression(network, optimizer='adam',loss='categorical_crossentropy',learning_rate=0.001);
#network = regression(network, optimizer= Optimizer(name='adam',learning_rate=0.00005),loss='categorical_crossentropy',learning_rate=0.001);

```model = tflearn.DNN(network,tensorboard_verbose=1,checkpoint_path=network_directory+"convolutional_nerve_identifier.ckpt",tensorboard_dir=models_path+'tensorboard'); #
model.fit(images_array,masks_array,n_epoch=epoch,batch_size=batch,show_metric=True);

model.save(network_directory+"CNN_YES_NO_CLASSIFIER.tfl");
tensorflow.reset_default_graph() ;

``````

[llealgt]

Well, debugging my training found that it is not converging, so im getting a Nan cost and that is causing the error, i'll try to lower my learning rate.

[bidai541]

I met the same problem and solved by decreasing the learning rate.

[491506870]

I also used the optimizer Adam, and when my learning rate was 0.001, I met this problem, then I turned my learning rate to 1e-5, it still had this problem, do you know why and what can I do? thank you!

[chenfengqing]

ultrasound_launcher.py
change trainer = unet.Trainer(net, norm_grads=True, optimizer="adam")
to trainer = unet.Trainer(net, norm_grads=True, optimizer="momentum"),
for now ,it works