Deepspeech: It seems that global step (tf.train.get_global_step) has not been increased

[luckmoon]

Here is my terminal log:

I --------------------------------------------------------------------------------
I WER: 1.000000, loss: 35.457664, mean edit distance: 0.909091
I  - src: "how are you"
I  - res: "i "
I --------------------------------------------------------------------------------
WARNING:tensorflow:It seems that global step (tf.train.get_global_step) has not been increased. Current value (could be stable): 250 vs previous value: 250. You could increase the global step by passing tf.train.get_global_step() to Optimizer.apply_gradients or Optimizer.minimize.
WARNING:tensorflow:It seems that global step (tf.train.get_global_step) has not been increased. Current value (could be stable): 250 vs previous value: 250. You could increase the global step by passing tf.train.get_global_step() to Optimizer.apply_gradients or Optimizer.minimize.
WARNING:tensorflow:It seems that global step (tf.train.get_global_step) has not been increased. Current value (could be stable): 250 vs previous value: 250. You could increase the global step by passing tf.train.get_global_step() to Optimizer.apply_gradients or Optimizer.minimize.

And Warning that global step (tf.train.get_global_step) has not been increased ALL THE TIME, rather than exporting my expected model.
Maybe this is a BUG?

Below is my command：
CUDA_VISIBLE_DEVICES=0,1,2,3 ./DeepSpeech.py --train_files ../data/CV/cv-valid-train.csv --dev_files ../data/CV/cv-valid-dev.csv --test_files ../data/CV/cv-valid-test.csv --epoch 2 --limit_train 1000 --checkpoint_dir './checkpoint4' --export_dir './export_models' --n_hidden 64 --display_step 1 --validation_step 1

[philofellow]

I'm having the same problem. After 20 epochs, I got:
WARNING:tensorflow:It seems that global step (tf.train.get_global_step) has not been increased. Current value (could be stable): 600 vs previous value: 600. You could increase the global step by passing tf.train.get_global_step() to Optimizer.apply_gradients or Optimizer.minimize.
I Test of Epoch 20 - WER: 0.978333, loss: 96.74490496317546, mean edit distance: 0.924535
I --------------------------------------------------------------------------------
I WER: 0.750000, loss: 68.334908, mean edit distance: 0.857143
I - src: "what can i say"
I - res: "i "
I --------------------------------------------------------------------------------
I WER: 1.000000, loss: 57.347744, mean edit distance: 0.916667
I - src: "are you open"
I - res: "i "

[philofellow]

btw, my training set only contains 30 audio clips. will that be a problem?

[lissyx]

@philofellow It's happening during validation step, not training ?

[philofellow]

you are right. it is during validation. so that is not a problem. thanks.

[daanzu]

I'm having (I think) the same issue: repeated warnings from TF about global step not being increased. They don't occur during training, only during validation and testing. It's not a breaking problem, just annoying.

A clear illustration:

I STARTING Optimization
I Training of Epoch 0 - loss: 32.377772
WARNING:tensorflow:It seems that global step (tf.train.get_global_step) has not been increased. Current value (could be stable): 166 vs previous value: 166. You could increase the global step by passing tf.train.get_global_step() to Optimizer.apply_gradients or Optimizer.minimize.
WARNING:tensorflow:It seems that global step (tf.train.get_global_step) has not been increased. Current value (could be stable): 166 vs previous value: 166. You could increase the global step by passing tf.train.get_global_step() to Optimizer.apply_gradients or Optimizer.minimize.
I Validation of Epoch 0 - loss: 26.880103
I Training of Epoch 1 - loss: 28.446264
WARNING:tensorflow:It seems that global step (tf.train.get_global_step) has not been increased. Current value (could be stable): 332 vs previous value: 332. You could increase the global step by passing tf.train.get_global_step() to Optimizer.apply_gradients or Optimizer.minimize.
WARNING:tensorflow:It seems that global step (tf.train.get_global_step) has not been increased. Current value (could be stable): 332 vs previous value: 332. You could increase the global step by passing tf.train.get_global_step() to Optimizer.apply_gradients or Optimizer.minimize.
I Validation of Epoch 1 - loss: 27.169247
I Training of Epoch 2 - loss: 25.259514
WARNING:tensorflow:It seems that global step (tf.train.get_global_step) has not been increased. Current value (could be stable): 498 vs previous value: 498. You could increase the global step by passing tf.train.get_global_step() to Optimizer.apply_gradients or Optimizer.minimize.
WARNING:tensorflow:It seems that global step (tf.train.get_global_step) has not been increased. Current value (could be stable): 498 vs previous value: 498. You could increase the global step by passing tf.train.get_global_step() to Optimizer.apply_gradients or Optimizer.minimize.
I Validation of Epoch 2 - loss: 22.037639
[snip]
I Training of Epoch 59 - loss: 0.554116
WARNING:tensorflow:It seems that global step (tf.train.get_global_step) has not been increased. Current value (could be stable): 9960 vs previous value: 9960. You could increase the global step by passing tf.train.get_global_step() to Optimizer.apply_gradients or Optimizer.minimize.
WARNING:tensorflow:It seems that global step (tf.train.get_global_step) has not been increased. Current value (could be stable): 9960 vs previous value: 9960. You could increase the global step by passing tf.train.get_global_step() to Optimizer.apply_gradients or Optimizer.minimize.
I Validation of Epoch 59 - loss: 1.535293
I FINISHED Optimization - training time: 0:21:16
WARNING:tensorflow:It seems that global step (tf.train.get_global_step) has not been increased. Current value (could be stable): 9960 vs previous value: 9960. You could increase the global step by passing tf.train.get_global_step() to Optimizer.apply_gradients or Optimizer.minimize.
I Test of Epoch 60 - WER: 0.026473, loss: 1.072239521580438, mean edit distance: 0.018174
I --------------------------------------------------------------------------------
[snip]

[reuben]

1732 should fix this.

[lock[bot]]

This thread has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.