Tensorboard: tensorflow beta0 or 1, tensorboard: Windows fatal exception: access violation

[duancaohui]

Environment information (required)

Windows 10
RTX 2080 Ti
tensortlow-gpu== 2.0.0beta1
CUDA 10.0, CUDNN cudnn-10.0-windows10-x64-v7.4.2.24

Issue description

When I update my tensorflow-gpu==alpha0 to tensorflow-gpu== 2.0.0beta1，I open tensorboard to view my training process,

I:\Tensorflow\pix2pix\train_logs\20190628-100842>tensorboard --logdir train
TensorBoard 1.14.0a20190603 at http://DESKTOP-K1IS85B:6006/ (Press CTRL+C to quit)

However, when I open the chrome and visit the site, a mistake arised as follows:

`Windows fatal exception: access violation

Thread 0x00004044 (most recent call first):
  File "c:\users\admin\appdata\local\programs\python\python36\lib\site-packages\tensorflow\python\lib\io\file_io.py", line 596 in is_directory_v2
  File "c:\users\admin\appdata\local\programs\python\python36\lib\site-packages\tensorflow\python\lib\io\file_io.py", line 582 in is_directory
  File "c:\users\admin\appdata\local\programs\python\python36\lib\site-packages\tensorflow\python\lib\io\file_io.py", line 634 in list_directory_v2
  File "c:\users\admin\appdata\local\programs\python\python36\lib\site-packages\tensorboard\backend\event_processing\plugin_asset_util.py", line 76 in ListAssets
  File "c:\users\admin\appdata\local\programs\python\python36\lib\site-packages\tensorboard\backend\event_processing\plugin_event_accumulator.py", line 191 in PluginAssets
  File "c:\users\admin\appdata\local\programs\python\python36\lib\site-packages\tensorboard\backend\event_processing\plugin_event_multiplexer.py", line 263 in <dictcomp>
  File "c:\users\admin\appdata\local\programs\python\python36\lib\site-packages\tensorboard\backend\event_processing\plugin_event_multiplexer.py", line 263 in PluginAssets
  File "c:\users\admin\appdata\local\programs\python\python36\lib\site-packages\tensorboard\plugins\profile\profile_plugin.py", line 251 in generate_run_to_tools
  File "c:\users\admin\appdata\local\programs\python\python36\lib\site-packages\tensorboard\plugins\profile\profile_plugin.py", line 130 in compute_is_active
  File "c:\users\admin\appdata\local\programs\python\python36\lib\threading.py", line 864 in run
  File "c:\users\admin\appdata\local\programs\python\python36\lib\threading.py", line 916 in _bootstrap_inner
  File "c:\users\admin\appdata\local\programs\python\python36\lib\threading.py", line 884 in _bootstrap

Thread 0x000015f4 (most recent call first):
  File "c:\users\admin\appdata\local\programs\python\python36\lib\socket.py", line 586 in readinto
  File "c:\users\admin\appdata\local\programs\python\python36\lib\site-packages\werkzeug\serving.py", line 324 in handle_one_request
  File "c:\users\admin\appdata\local\programs\python\python36\lib\http\server.py", line 418 in handle
  File "c:\users\admin\appdata\local\programs\python\python36\lib\site-packages\werkzeug\serving.py", line 293 in handle
  File "c:\users\admin\appdata\local\programs\python\python36\lib\socketserver.py", line 721 in __init__
  File "c:\users\admin\appdata\local\programs\python\python36\lib\socketserver.py", line 361 in finish_request
  File "c:\users\admin\appdata\local\programs\python\python36\lib\socketserver.py", line 651 in process_request_thread
  File "c:\users\admin\appdata\local\programs\python\python36\lib\threading.py", line 864 in run
  File "c:\users\admin\appdata\local\programs\python\python36\lib\threading.py", line 916 in _bootstrap_inner
  File "c:\users\admin\appdata\local\programs\python\python36\lib\threading.py", line 884 in _bootstrap

Thread 0x00003f70 (most recent call first):
  File "c:\users\admin\appdata\local\programs\python\python36\lib\socket.py", line 586 in readinto
  File "c:\users\admin\appdata\local\programs\python\python36\lib\site-packages\werkzeug\serving.py", line 324 in handle_one_request
  File "c:\users\admin\appdata\local\programs\python\python36\lib\http\server.py", line 418 in handle
  File "c:\users\admin\appdata\local\programs\python\python36\lib\site-packages\werkzeug\serving.py", line 293 in handle
  File "c:\users\admin\appdata\local\programs\python\python36\lib\socketserver.py", line 721 in __init__
  File "c:\users\admin\appdata\local\programs\python\python36\lib\socketserver.py", line 361 in finish_request
  File "c:\users\admin\appdata\local\programs\python\python36\lib\socketserver.py", line 651 in process_request_thread
  File "c:\users\admin\appdata\local\programs\python\python36\lib\threading.py", line 864 in run
  File "c:\users\admin\appdata\local\programs\python\python36\lib\threading.py", line 916 in _bootstrap_inner
  File "c:\users\admin\appdata\local\programs\python\python36\lib\threading.py", line 884 in _bootstrap

Current thread 0x00001284 (most recent call first):
  File "c:\users\admin\appdata\local\programs\python\python36\lib\site-packages\tensorflow\python\lib\io\file_io.py", line 84 in _preread_check
  File "c:\users\admin\appdata\local\programs\python\python36\lib\site-packages\tensorflow\python\lib\io\file_io.py", line 122 in read
  File "c:\users\admin\appdata\local\programs\python\python36\lib\site-packages\tensorflow\python\lib\io\file_io.py", line 333 in read_file_to_string
  File "c:\users\admin\appdata\local\programs\python\python36\lib\site-packages\tensorflow\python\training\checkpoint_management.py", line 273 in get_checkpoint_state
  File "c:\users\admin\appdata\local\programs\python\python36\lib\site-packages\tensorflow\python\training\checkpoint_management.py", line 336 in latest_checkpoint
  File "c:\users\admin\appdata\local\programs\python\python36\lib\site-packages\tensorboard\plugins\projector\projector_plugin.py", line 663 in _find_latest_checkpoint
  File "c:\users\admin\appdata\local\programs\python\python36\lib\site-packages\tensorboard\plugins\projector\projector_plugin.py", line 403 in _read_latest_config_files
  File "c:\users\admin\appdata\local\programs\python\python36\lib\site-packages\tensorboard\plugins\projector\projector_plugin.py", line 323 in configs
  File "c:\users\admin\appdata\local\programs\python\python36\lib\site-packages\tensorboard\plugins\projector\projector_plugin.py", line 306 in _determine_is_active
  File "c:\users\admin\appdata\local\programs\python\python36\lib\threading.py", line 864 in run
  File "c:\users\admin\appdata\local\programs\python\python36\lib\threading.py", line 916 in _bootstrap_inner
  File "c:\users\admin\appdata\local\programs\python\python36\lib\threading.py", line 884 in _bootstrap

Thread 0x000014e0 (most recent call first):
  File "c:\users\admin\appdata\local\programs\python\python36\lib\socket.py", line 586 in readinto
  File "c:\users\admin\appdata\local\programs\python\python36\lib\site-packages\werkzeug\serving.py", line 324 in handle_one_request
  File "c:\users\admin\appdata\local\programs\python\python36\lib\http\server.py", line 420 in handle
  File "c:\users\admin\appdata\local\programs\python\python36\lib\site-packages\werkzeug\serving.py", line 293 in handle
  File "c:\users\admin\appdata\local\programs\python\python36\lib\socketserver.py", line 721 in __init__
  File "c:\users\admin\appdata\local\programs\python\python36\lib\socketserver.py", line 361 in finish_request
  File "c:\users\admin\appdata\local\programs\python\python36\lib\socketserver.py", line 651 in process_request_thread
  File "c:\users\admin\appdata\local\programs\python\python36\lib\threading.py", line 864 in run
  File "c:\users\admin\appdata\local\programs\python\python36\lib\threading.py", line 916 in _bootstrap_inner
  File "c:\users\admin\appdata\local\programs\python\python36\lib\threading.py", line 884 in _bootstrap

Thread 0x00001428 (most recent call first):
  File "c:\users\admin\appdata\local\programs\python\python36\lib\threading.py", line 295 in wait
  File "c:\users\admin\appdata\local\programs\python\python36\lib\threading.py", line 551 in wait
  File "c:\users\admin\appdata\local\programs\python\python36\lib\threading.py", line 851 in start
  File "c:\users\admin\appdata\local\programs\python\python36\lib\site-packages\tensorboard\plugins\profile\profile_plugin.py", line 135 in is_active
  File "c:\users\admin\appdata\local\programs\python\python36\lib\site-packages\tensorboard\backend\application.py", line 286 in _serve_plugins_listing
  File "c:\users\admin\appdata\local\programs\python\python36\lib\site-packages\werkzeug\wrappers.py", line 308 in application
  File "c:\users\admin\appdata\local\programs\python\python36\lib\site-packages\tensorboard\backend\application.py", line 317 in __call__
  File "c:\users\admin\appdata\local\programs\python\python36\lib\site-packages\werkzeug\serving.py", line 258 in execute
  File "c:\users\admin\appdata\local\programs\python\python36\lib\site-packages\werkzeug\serving.py", line 270 in run_wsgi
  File "c:\users\admin\appdata\local\programs\python\python36\lib\site-packages\werkzeug\serving.py", line 328 in handle_one_request
  File "c:\users\admin\appdata\local\programs\python\python36\lib\http\server.py", line 420 in handle
  File "c:\users\admin\appdata\local\programs\python\python36\lib\site-packages\werkzeug\serving.py", line 293 in handle
  File "c:\users\admin\appdata\local\programs\python\python36\lib\socketserver.py", line 721 in __init__
  File "c:\users\admin\appdata\local\programs\python\python36\lib\socketserver.py", line 361 in finish_request
  File "c:\users\admin\appdata\local\programs\python\python36\lib\socketserver.py", line 651 in process_request_thread
  File "c:\users\admin\appdata\local\programs\python\python36\lib\threading.py", line 864 in run
  File "c:\users\admin\appdata\local\programs\python\python36\lib\threading.py", line 916 in _bootstrap_inner
  File "c:\users\admin\appdata\local\programs\python\python36\lib\threading.py", line 884 in _bootstrap

Thread 0x00000acc (most recent call first):
  File "c:\users\admin\appdata\local\programs\python\python36\lib\site-packages\tensorboard\backend\application.py", line 404 in _reload
  File "c:\users\admin\appdata\local\programs\python\python36\lib\threading.py", line 864 in run
  File "c:\users\admin\appdata\local\programs\python\python36\lib\threading.py", line 916 in _bootstrap_inner
  File "c:\users\admin\appdata\local\programs\python\python36\lib\threading.py", line 884 in _bootstrap

Thread 0x00003698 (most recent call first):
  File "c:\users\admin\appdata\local\programs\python\python36\lib\threading.py", line 295 in wait
  File "c:\users\admin\appdata\local\programs\python\python36\lib\threading.py", line 551 in wait
  File "c:\users\admin\appdata\local\programs\python\python36\lib\threading.py", line 851 in start
  File "c:\users\admin\appdata\local\programs\python\python36\lib\socketserver.py", line 666 in process_request
  File "c:\users\admin\appdata\local\programs\python\python36\lib\socketserver.py", line 317 in _handle_request_noblock
  File "c:\users\admin\appdata\local\programs\python\python36\lib\socketserver.py", line 238 in serve_forever
  File "c:\users\admin\appdata\local\programs\python\python36\lib\site-packages\werkzeug\serving.py", line 612 in serve_forever
  File "c:\users\admin\appdata\local\programs\python\python36\lib\site-packages\tensorboard\program.py", line 233 in main
  File "c:\users\admin\appdata\local\programs\python\python36\lib\site-packages\absl\app.py", line 251 in _run_main
  File "c:\users\admin\appdata\local\programs\python\python36\lib\site-packages\absl\app.py", line 300 in run
  File "c:\users\admin\appdata\local\programs\python\python36\lib\site-packages\tensorboard\main.py", line 64 in run_main
  File "C:\Users\Admin\AppData\Local\Programs\Python\Python36\Scripts\tensorboard.exe\__main__.py", line 9 in <module>
  File "c:\users\admin\appdata\local\programs\python\python36\lib\runpy.py", line 85 in _run_code
  File "c:\users\admin\appdata\local\programs\python\python36\lib\runpy.py", line 193 in _run_module_as_main

Therefore, I re-install the tensorflow-gpu==2.0.0alpha0, everything is all right.

[nfelt]

It sounds like reinstalling tensorflow-gpu fixed the issue, and it's not clear to me from the thread dump what the problem might have been, so I'll go ahead and close this out. If you end up seeing this repeatedly and can reproduce the issue from a clean install (see e.g. the issue instructions), feel free to comment and we can re-open.

[WuJunde]

I meet the similar problem and reinstalling doesn't work

`Windows fatal exception: access violation

Current thread 0x00001548 (most recent call first):
File "c:\programdata\anaconda3\lib\site-packagestensorflow\python\lib\io\file_io.py", line 384 in get_matching_files_v2
File "c:\programdata\anaconda3\lib\site-packagestensorflow\python\lib\io\file_io.py", line 363 in get_matching_files
File "c:\programdata\anaconda3\lib\site-packages\magenta\models\image_stylization\image_stylization_create_dataset.py", line 45 in _parse_style_files
File "c:\programdata\anaconda3\lib\site-packages\magenta\models\image_stylization\image_stylization_create_dataset.py", line 68 in main
File "c:\programdata\anaconda3\lib\site-packages\absl\app.py", line 251 in _run_main
File "c:\programdata\anaconda3\lib\site-packages\absl\app.py", line 300 in run
File "c:\programdata\anaconda3\lib\site-packagestensorflow\python\platform\app.py", line 40 in run
File "c:\programdata\anaconda3\lib\site-packages\magenta\models\image_stylization\image_stylization_create_dataset.py", line 100 in console_entry_point
File "C:\ProgramDataAnaconda3\Scripts\image_stylization_create_dataset.exe__main__.py", line 9 in
File "c:\programdata\anaconda3\librunpy.py", line 85 in _run_code
File "c:\programdata\anaconda3\librunpy.py", line 193 in _run_module_as_main`

Some more information:
It may seem have nothing to do with tensorboard, but wirely, just before it happened I met the error:
https://github.com/tensorflow/tensorboard/issues/1433
I upgraded my absl under this environment, and it didn't work. I then find the file actually linked to the absl in an another environment (don't know why) . So I upgraded absl in all my environments. Then I re-run it and this error occured.

Does anyone have any advice? thank you

[wchargin]

Hi @WuJunde—it looks like all the potentially problematic stack frames
are in TensorFlow’s file_io.py. (The only TensorBoard stack frame is
in a call to sleep, and the rest are from the Python standard
library.)

Could you please report this as a TensorFlow bug?

[dasmehdix]

I had same problem and solved it.Problem was occured when I tried to transfer learning with faster_rcnn_inception_v2_coco model.

As far as I understand the problem about the config file of the model you try to use.First check the path of config file correct or wrong.Then, check the paths given in the config file.I solved the problem by correct the paths.

[loliq]

Hi

I had same problem and solved it.Problem was occured when I tried to transfer learning with faster_rcnn_inception_v2_coco model.

As far as I understand the problem about the config file of the model you try to use.First check the path of config file correct or wrong.Then, check the paths given in the config file.I solved the problem by correct the paths.

Hi, I met the same problem using object detection api. And I check the path, but it didn't work, my tensorflow version is 1.14.

[dasmehdix]

Hi

I had same problem and solved it.Problem was occured when I tried to transfer learning with faster_rcnn_inception_v2_coco model.
As far as I understand the problem about the config file of the model you try to use.First check the path of config file correct or wrong.Then, check the paths given in the config file.I solved the problem by correct the paths.

Hi, I met the same problem using object detection api. And I check the path, but it didn't work, my tensorflow version is 1.14.

Try to erase the codes below in your config file.Check num_examples and num_classes in your config file too.

schedule {
            step: 0
            learning_rate: .0001
          }

[cnzjhdx]

Hi @WuJunde—it looks like all the potentially problematic stack frames
are in TensorFlow’s file_io.py. (The only TensorBoard stack frame is
in a call to sleep, and the rest are from the Python standard
library.)

Could you please report this as a TensorFlow bug?
I report a bug,pls fixe.that's really problem for windows users.
Windows fatal exception: access violation #35777

[faridomarzadeh]

same problem here

[zhang12300]

same problem here 555

[zhang12300]

I found a strange thing that if I delete the 'checkpoint' directory or change its name to anything else even 'checkpoints', the problem will disappear.

[ColbertBrave]

I changed my tensorboard version to 1.6.0, then the problem disappeared.