Models: what is global step and loss?

[ghost]

Can some body explain for me about:
1- global step in the output of object detection API:
Is global step the iteration? does it refer to the batch?
and what is the difference of global step/sec and sec/step?

2- Loss:
Whats is exactly loss?
Is loss sum of localization loss and classification loss?

[derekjchow]

1. global_step is the iteration (so global_step 50 means we're processing the 50th batch). global step/sec and sec/step represent the same thing (one is the inverse of the other).

2. Correct. loss represents the sum of localization and classification losses.

[swg209]

Hi, I'd like to know at the same global_step have different loss ,what does this mean?
eg:
INFO:tensorflow:global step 58: loss = 7.3782 (1.219 sec/step)
INFO:tensorflow:global step 58: loss = 7.3782 (1.219 sec/step)
INFO:tensorflow:global step 58: loss = 7.4884 (1.265 sec/step)
INFO:tensorflow:global step 58: loss = 7.4884 (1.265 sec/step)
INFO:tensorflow:global step 58: loss = 7.4400 (1.251 sec/step)
INFO:tensorflow:global step 58: loss = 7.4400 (1.251 sec/step)
INFO:tensorflow:global step 58: loss = 7.6768 (1.257 sec/step)
INFO:tensorflow:global step 58: loss = 7.6768 (1.257 sec/step)
INFO:tensorflow:global step 58: loss = 7.3657 (1.219 sec/step)
INFO:tensorflow:global step 58: loss = 7.3657 (1.219 sec/step)
INFO:tensorflow:global step 58: loss = 7.7646 (1.238 sec/step)
INFO:tensorflow:global step 58: loss = 7.7646 (1.238 sec/step)

[thanhdochi]

I think for some reason, tensorflow get error when train that step and it rety couple of time if it can get over that issue. So it makes tf train couple of time on the same batch.

[omerrciftcii]

I think for some reason, tensorflow get error when train that step and it rety couple of time if it can get over that issue. So it makes tf train couple of time on the same batch.

Hello, did you find an answer?

[Mubashshira]

INFO:tensorflow:Restoring parameters from C:/tensorflowf/model/research/object_detection/faster_rcnn_inception_v2_coco_2018_01_28/model.ckpt
INFO:tensorflow:Restoring parameters from C:/tensorflowf/model/research/object_detection/faster_rcnn_inception_v2_coco_2018_01_28/model.ckpt
INFO:tensorflow:Running local_init_op.
INFO:tensorflow:Running local_init_op.
INFO:tensorflow:Done running local_init_op.
INFO:tensorflow:Done running local_init_op.
INFO:tensorflow:Starting Session.
INFO:tensorflow:Starting Session.
INFO:tensorflow:Saving checkpoint to path training/model.ckpt
INFO:tensorflow:Saving checkpoint to path training/model.ckpt
INFO:tensorflow:Starting Queues.
INFO:tensorflow:Starting Queues.
INFO:tensorflow:global_step/sec: 0
INFO:tensorflow:global_step/sec: 0
2019-06-14 12:12:22.930458: I tensorflow/core/kernels/data/shuffle_dataset_op.cc:101] Filling up shuffle buffer (this may take a while): 33 of 2048
2019-06-14 12:12:28.980816: I tensorflow/core/kernels/data/shuffle_dataset_op.cc:101] Filling up shuffle buffer (this may take a while): 61 of 2048
2019-06-14 12:12:45.020111: I tensorflow/core/kernels/data/shuffle_dataset_op.cc:101] Filling up shuffle buffer (this may take a while): 213 of 2048
2019-06-14 12:12:51.291855: I tensorflow/core/kernels/data/shuffle_dataset_op.cc:101] Filling up shuffle buffer (this may take a while): 214 of 2048
2019-06-14 12:13:23.427986: I tensorflow/core/kernels/data/shuffle_dataset_op.cc:101] Filling up shuffle buffer (this may take a while): 241 of 2048
2019-06-14 12:13:25.456611: I tensorflow/core/kernels/data/shuffle_dataset_op.cc:101] Filling up shuffle buffer (this may take a while): 242 of 2048
2019-06-14 12:13:25.471453: I tensorflow/core/kernels/data/shuffle_dataset_op.cc:101] Filling up shuffle buffer (this may take a while): 243 of 2048
2019-06-14 12:13:33.336883: I tensorflow/core/kernels/data/shuffle_dataset_op.cc:101] Filling up shuffle buffer (this may take a while): 361 of 2048
2019-06-14 12:13:38.762440: I tensorflow/core/kernels/data/shuffle_dataset_op.cc:101] Filling up shuffle buffer (this may take a while): 379 of 2048
2019-06-14 12:13:49.382400: I tensorflow/core/kernels/data/shuffle_dataset_op.cc:101] Filling up shuffle buffer (this may take a while): 401 of 2048
2019-06-14 12:13:58.524647: I tensorflow/core/kernels/data/shuffle_dataset_op.cc:101] Filling up shuffle buffer (this may take a while): 417 of 2048
2019-06-14 12:14:08.702036: I tensorflow/core/kernels/data/shuffle_dataset_op.cc:101] Filling up shuffle buffer (this may take a while): 551 of 2048
2019-06-14 12:14:26.377263: I tensorflow/core/kernels/data/shuffle_dataset_op.cc:101] Filling up shuffle buffer (this may take a while): 562 of 2048
2019-06-14 12:14:28.824022: I tensorflow/core/kernels/data/shuffle_dataset_op.cc:101] Filling up shuffle buffer (this may take a while): 563 of 2048
2019-06-14 12:14:38.586723: I tensorflow/core/kernels/data/shuffle_dataset_op.cc:101] Filling up shuffle buffer (this may take a while): 577 of 2048
2019-06-14 12:22:22.802403: I tensorflow/core/kernels/data/shuffle_dataset_op.cc:101] Filling up shuffle buffer (this may take a while): 578 of 2048
2019-06-14 12:22:27.472200: I tensorflow/core/kernels/data/shuffle_dataset_op.cc:101] Filling up shuffle buffer (this may take a while): 579 of 2048
2019-06-14 12:22:27.505150: I tensorflow/core/kernels/data/shuffle_dataset_op.cc:101] Filling up shuffle buffer (this may take a while): 580 of 2048
2019-06-14 12:22:27.526439: I tensorflow/core/kernels/data/shuffle_dataset_op.cc:101] Filling up shuffle buffer (this may take a while): 581 of 2048
2019-06-14 12:22:27.539010: I tensorflow/core/kernels/data/shuffle_dataset_op.cc:101] Filling up shuffle buffer (this may take a while): 582 of 2048
2019-06-14 12:22:27.553322: I tensorflow/core/kernels/data/shuffle_dataset_op.cc:101] Filling up shuffle buffer (this may take a while): 583 of 2048
2019-06-14 12:22:27.565258: I tensorflow/core/kernels/data/shuffle_dataset_op.cc:101] Filling up shuffle buffer (this may take a while): 584 of 2048
2019-06-14 12:22:27.577744: I tensorflow/core/kernels/data/shuffle_dataset_op.cc:101] Filling up shuffle buffer (this may take a while): 585 of 2048
2019-06-14 12:22:27.588644: I tensorflow/core/kernels/data/shuffle_dataset_op.cc:101] Filling up shuffle buffer (this may take a while): 586 of 2048
2019-06-14 12:22:28.782630: I tensorflow/core/kernels/data/shuffle_dataset_op.cc:101] Filling up shuffle buffer (this may take a while): 587 of 2048
2019-06-14 12:22:28.810710: I tensorflow/core/kernels/data/shuffle_dataset_op.cc:101] Filling up shuffle buffer (this may take a while): 588 of 2048
2019-06-14 12:22:28.838413: I tensorflow/core/kernels/data/shuffle_dataset_op.cc:101] Filling up shuffle buffer (this may take a while): 589 of 2048
2019-06-14 12:22:28.858302: I tensorflow/core/kernels/data/shuffle_dataset_op.cc:101] Filling up shuffle buffer (this may take a while): 590 of 2048
2019-06-14 12:22:28.872743: I tensorflow/core/kernels/data/shuffle_dataset_op.cc:101] Filling up shuffle buffer (this may take a while): 591 of 2048
2019-06-14 12:22:28.885641: I tensorflow/core/kernels/data/shuffle_dataset_op.cc:101] Filling up shuffle buffer (this may take a while): 592 of 2048
2019-06-14 12:22:28.900502: I tensorflow/core/kernels/data/shuffle_dataset_op.cc:101] Filling up shuffle buffer (this may take a while): 593 of 2048
2019-06-14 12:22:28.914004: I tensorflow/core/kernels/data/shuffle_dataset_op.cc:101] Filling up shuffle buffer (this may take a while): 594 of 2048
2019-06-14 12:22:28.927656: I tensorflow/core/kernels/data/shuffle_dataset_op.cc:101] Filling up shuffle buffer (this may take a while): 595 of 2048
2019-06-14 12:22:28.935674: I tensorflow/core/kernels/data/shuffle_dataset_op.cc:101] Filling up shuffle buffer (this may take a while): 596 of 2048
2019-06-14 12:22:28.980765: I tensorflow/core/kernels/data/shuffle_dataset_op.cc:101] Filling up shuffle buffer (this may take a while): 597 of 2048
2019-06-14 12:22:29.835328: I tensorflow/core/kernels/data/shuffle_dataset_op.cc:101] Filling up shuffle buffer (this may take a while): 598 of 2048
2019-06-14 12:22:29.870276: I tensorflow/core/kernels/data/shuffle_dataset_op.cc:101] Filling up shuffle buffer (this may take a while): 599 of 2048
2019-06-14 12:22:29.893878: I tensorflow/core/kernels/data/shuffle_dataset_op.cc:101] Filling up shuffle buffer (this may take a while): 600 of 2048
2019-06-14 12:22:44.594495: I tensorflow/core/kernels/data/shuffle_dataset_op.cc:101] Filling up shuffle buffer (this may take a while): 601 of 2048
2019-06-14 12:22:46.670034: I tensorflow/core/kernels/data/shuffle_dataset_op.cc:101] Filling up shuffle buffer (this may take a while): 602 of 2048
2019-06-14 12:22:46.689540: I tensorflow/core/kernels/data/shuffle_dataset_op.cc:101] Filling up shuffle buffer (this may take a while): 603 of 2048
2019-06-14 12:22:50.023627: I tensorflow/core/kernels/data/shuffle_dataset_op.cc:101] Filling up shuffle buffeforrtl: error (200): program aborting due to control-C event
Image PC Routine Line Source
libifcoremd.dll 00007FF834693B58 Unknown Unknown Unknown
KERNELBASE.dll 00007FF86AFD56FD Unknown Unknown Unknown
KERNEL32.DLL 00007FF86C104034 Unknown Unknown Unknown
ntdll.dll 00007FF86EAB3691 Unknown Unknown Unknown

i m not getting these global step

[tensorflowbutler]

Hi There,
We are checking to see if you still need help on this, as this seems to be considerably old issue. Please update this issue with the latest information, code snippet to reproduce your issue and error you are seeing.
If we don't hear from you in the next 7 days, this issue will be closed automatically. If you don't need help on this issue any more, please consider closing this.

[mychen]

Hi all,

I also have the same problem.
INFO:tensorflow:Recording summary at step 0.
INFO:tensorflow:Recording summary at step 0.
INFO:tensorflow:global step 1: loss = 79.3501 (57.126 sec/step)
INFO:tensorflow:global step 1: loss = 79.3501 (57.126 sec/step)
INFO:tensorflow:global step 2: loss = 72.8609 (17.053 sec/step)
INFO:tensorflow:global step 2: loss = 72.8609 (17.053 sec/step)
INFO:tensorflow:global step 3: loss = 67.3308 (16.534 sec/step)
INFO:tensorflow:global step 3: loss = 67.3308 (16.534 sec/step)
INFO:tensorflow:global step 4: loss = 64.2655 (16.926 sec/step)
INFO:tensorflow:global step 4: loss = 64.2655 (16.926 sec/step)
INFO:tensorflow:Recording summary at step 4.
INFO:tensorflow:Recording summary at step 4.
INFO:tensorflow:global_step/sec: 0.0339311
INFO:tensorflow:global_step/sec: 0.0339311
INFO:tensorflow:global step 5: loss = 62.5264 (20.205 sec/step)
INFO:tensorflow:global step 5: loss = 62.5264 (20.205 sec/step)
INFO:tensorflow:global step 6: loss = 60.8808 (17.060 sec/step)
INFO:tensorflow:global step 6: loss = 60.8808 (17.060 sec/step)
INFO:tensorflow:global step 7: loss = 59.8181 (16.456 sec/step)
INFO:tensorflow:global step 7: loss = 59.8181 (16.456 sec/step)
INFO:tensorflow:global step 8: loss = 57.9715 (16.765 sec/step)
INFO:tensorflow:global step 8: loss = 57.9715 (16.765 sec/step)
INFO:tensorflow:global step 9: loss = 57.1203 (16.756 sec/step)
INFO:tensorflow:global step 9: loss = 57.1203 (16.756 sec/step)
INFO:tensorflow:global step 10: loss = 56.4704 (16.407 sec/step)
INFO:tensorflow:global step 10: loss = 56.4704 (16.407 sec/step)
INFO:tensorflow:global step 11: loss = 55.3032 (16.196 sec/step)
INFO:tensorflow:global step 11: loss = 55.3032 (16.196 sec/step)
INFO:tensorflow:Recording summary at step 11.
INFO:tensorflow:Recording summary at step 11.
INFO:tensorflow:global step 12: loss = 54.4417 (18.842 sec/step)
INFO:tensorflow:global step 12: loss = 54.4417 (18.842 sec/step)
INFO:tensorflow:global_step/sec: 0.0666667
INFO:tensorflow:global_step/sec: 0.0666667
INFO:tensorflow:global step 13: loss = 53.3713 (16.639 sec/step)
INFO:tensorflow:global step 13: loss = 53.3713 (16.639 sec/step)
INFO:tensorflow:global step 14: loss = 51.8879 (18.102 sec/step)
INFO:tensorflow:global step 14: loss = 51.8879 (18.102 sec/step)
INFO:tensorflow:global step 15: loss = 51.4940 (21.297 sec/step)
INFO:tensorflow:global step 15: loss = 51.4940 (21.297 sec/step)
INFO:tensorflow:global step 16: loss = 50.9731 (17.981 sec/step)
INFO:tensorflow:global step 16: loss = 50.9731 (17.981 sec/step)
INFO:tensorflow:global step 17: loss = 50.4010 (18.549 sec/step)
INFO:tensorflow:global step 17: loss = 50.4010 (18.549 sec/step)
INFO:tensorflow:global step 18: loss = 49.2269 (18.066 sec/step)
INFO:tensorflow:global step 18: loss = 49.2269 (18.066 sec/step)
INFO:tensorflow:Recording summary at step 18.
INFO:tensorflow:Recording summary at step 18.
INFO:tensorflow:global_step/sec: 0.05
INFO:tensorflow:global_step/sec: 0.05
INFO:tensorflow:global step 19: loss = 48.4056 (20.208 sec/step)
INFO:tensorflow:global step 19: loss = 48.4056 (20.208 sec/step)
INFO:tensorflow:global step 20: loss = 47.7406 (17.664 sec/step)
INFO:tensorflow:global step 20: loss = 47.7406 (17.664 sec/step)
INFO:tensorflow:global step 21: loss = 45.2973 (18.819 sec/step)
INFO:tensorflow:global step 21: loss = 45.2973 (18.819 sec/step)
INFO:tensorflow:global step 22: loss = 46.2185 (17.941 sec/step)
INFO:tensorflow:global step 22: loss = 46.2185 (17.941 sec/step)
INFO:tensorflow:global step 23: loss = 46.4643 (17.777 sec/step)
INFO:tensorflow:global step 23: loss = 46.4643 (17.777 sec/step)
INFO:tensorflow:global step 24: loss = 44.8898 (19.788 sec/step)
INFO:tensorflow:global step 24: loss = 44.8898 (19.788 sec/step)
INFO:tensorflow:global_step/sec: 0.05
INFO:tensorflow:global_step/sec: 0.05
INFO:tensorflow:Recording summary at step 24.
INFO:tensorflow:Recording summary at step 24.
INFO:tensorflow:global step 25: loss = 44.3813 (23.713 sec/step)
INFO:tensorflow:global step 25: loss = 44.3813 (23.713 sec/step)
INFO:tensorflow:global step 26: loss = 43.4742 (22.435 sec/step)
INFO:tensorflow:global step 26: loss = 43.4742 (22.435 sec/step)
INFO:tensorflow:global step 27: loss = 44.0719 (19.627 sec/step)
INFO:tensorflow:global step 27: loss = 44.0719 (19.627 sec/step)
INFO:tensorflow:global step 28: loss = 42.0462 (19.540 sec/step)
INFO:tensorflow:global step 28: loss = 42.0462 (19.540 sec/step)
INFO:tensorflow:global step 29: loss = 42.9915 (17.847 sec/step)
INFO:tensorflow:global step 29: loss = 42.9915 (17.847 sec/step)
INFO:tensorflow:global step 30: loss = 41.6660 (21.918 sec/step)
INFO:tensorflow:global step 30: loss = 41.6660 (21.918 sec/step)
INFO:tensorflow:Saving checkpoint to path OD_training/model.ckpt
INFO:tensorflow:Saving checkpoint to path OD_training/model.ckpt
INFO:tensorflow:global_step/sec: 0.0498167
INFO:tensorflow:global_step/sec: 0.0498167
INFO:tensorflow:Recording summary at step 30.
INFO:tensorflow:Recording summary at step 30.
INFO:tensorflow:global step 31: loss = 40.8092 (25.096 sec/step)
INFO:tensorflow:global step 31: loss = 40.8092 (25.096 sec/step)
INFO:tensorflow:global step 32: loss = 39.4991 (20.044 sec/step)
INFO:tensorflow:global step 32: loss = 39.4991 (20.044 sec/step)
INFO:tensorflow:global step 33: loss = 39.5772 (19.805 sec/step)
INFO:tensorflow:global step 33: loss = 39.5772 (19.805 sec/step)
INFO:tensorflow:global step 34: loss = 39.0651 (21.057 sec/step)
INFO:tensorflow:global step 34: loss = 39.0651 (21.057 sec/step)
INFO:tensorflow:global step 35: loss = 37.8860 (19.396 sec/step)
INFO:tensorflow:global step 35: loss = 37.8860 (19.396 sec/step)
INFO:tensorflow:global step 36: loss = 38.1005 (19.428 sec/step)
INFO:tensorflow:global step 36: loss = 38.1005 (19.428 sec/step)
INFO:tensorflow:global_step/sec: 0.0501773
INFO:tensorflow:global_step/sec: 0.0501773
INFO:tensorflow:Recording summary at step 36.
INFO:tensorflow:Recording summary at step 36.
INFO:tensorflow:global step 37: loss = 38.0518 (22.693 sec/step)
INFO:tensorflow:global step 37: loss = 38.0518 (22.693 sec/step)
INFO:tensorflow:global step 38: loss = 36.6147 (19.496 sec/step)
INFO:tensorflow:global step 38: loss = 36.6147 (19.496 sec/step)
INFO:tensorflow:global step 39: loss = 36.3582 (18.432 sec/step)
INFO:tensorflow:global step 39: loss = 36.3582 (18.432 sec/step)
INFO:tensorflow:global step 40: loss = 35.3704 (19.956 sec/step)
INFO:tensorflow:global step 40: loss = 35.3704 (19.956 sec/step)
INFO:tensorflow:global step 41: loss = 36.0097 (19.789 sec/step)
INFO:tensorflow:global step 41: loss = 36.0097 (19.789 sec/step)
2020-02-07 17:09:38.256759: W tensorflow/core/framework/allocator.cc:124] Allocation of 1445222400 exceeds 10% of system memory.
INFO:tensorflow:global step 42: loss = 36.1353 (19.666 sec/step)
INFO:tensorflow:global step 42: loss = 36.1353 (19.666 sec/step)
2020-02-07 17:09:57.921058: W tensorflow/core/framework/allocator.cc:124] Allocation of 1328486400 exceeds 10% of system memory.
INFO:tensorflow:global_step/sec: 0.0498332
INFO:tensorflow:global_step/sec: 0.0498332
INFO:tensorflow:Recording summary at step 42.
INFO:tensorflow:Recording summary at step 42.
INFO:tensorflow:global step 43: loss = 34.0621 (20.643 sec/step)
INFO:tensorflow:global step 43: loss = 34.0621 (20.643 sec/step)
INFO:tensorflow:global step 44: loss = 33.6678 (19.522 sec/step)
INFO:tensorflow:global step 44: loss = 33.6678 (19.522 sec/step)
INFO:tensorflow:global step 45: loss = 34.3040 (19.075 sec/step)
INFO:tensorflow:global step 45: loss = 34.3040 (19.075 sec/step)
INFO:tensorflow:global step 46: loss = 34.3227 (19.133 sec/step)
INFO:tensorflow:global step 46: loss = 34.3227 (19.133 sec/step)
INFO:tensorflow:global step 47: loss = 36.0210 (18.916 sec/step)
INFO:tensorflow:global step 47: loss = 36.0210 (18.916 sec/step)
INFO:tensorflow:global step 48: loss = 33.0603 (18.288 sec/step)
INFO:tensorflow:global step 48: loss = 33.0603 (18.288 sec/step)
INFO:tensorflow:Recording summary at step 48.
INFO:tensorflow:Recording summary at step 48.
INFO:tensorflow:global_step/sec: 0.0501753
INFO:tensorflow:global_step/sec: 0.0501753
INFO:tensorflow:global step 49: loss = 32.6435 (20.826 sec/step)
INFO:tensorflow:global step 49: loss = 32.6435 (20.826 sec/step)
INFO:tensorflow:global step 50: loss = 33.6923 (16.848 sec/step)
INFO:tensorflow:global step 50: loss = 33.6923 (16.848 sec/step)
INFO:tensorflow:global step 51: loss = 31.5511 (17.574 sec/step)
INFO:tensorflow:global step 51: loss = 31.5511 (17.574 sec/step)