Tensorboard: Error using TensorBoard in a Jupyter Notebook on the Google Cloud Platform

[psyyip]

I followed instructions to launch TensorBoard like this:

`get_ipython().system_raw(
"tensorboard --logdir {} --host 0.0.0.0 --port 6006 &"
.format(OUTDIR)
)

get_ipython().system_raw("./assets/ngrok http 6006 &")

!curl -s http://localhost:4040/api/tunnels | python3 -c \
"import sys, json; print(json.load(sys.stdin)['tunnels'][0]['public_url'])"
`

The following error messages show up:

Traceback (most recent call last):
File "", line 1, in
File "/usr/lib/python3.5/json/init.py", line 268, in load
parse_constant=parse_constant, object_pairs_hook=object_pairs_hook, **kw)
File "/usr/lib/python3.5/json/init.py", line 319, in loads
return _default_decoder.decode(s)
File "/usr/lib/python3.5/json/decoder.py", line 339, in decode
obj, end = self.raw_decode(s, idx=_w(s, 0).end())
File "/usr/lib/python3.5/json/decoder.py", line 357, in raw_decode
raise JSONDecodeError("Expecting value", s, err.value) from None
json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)

Before running this code, I had already trained a Tensorflow model and exported it to the directory "OUTDIR". The tunnel connection is not set up and therefore no public URL is generated.

Can anyone kindly give me some advice as to how to solve it?
Thank you very much!

[psyyip]

Anyone can provide some advice to me?

[krinkere]

I had similar issue and it seemed that it had to do with ngrok that I had in assets folder being corrupted or had wrong permissions perhaps? Anyway, I got around it by downloading ngrok and i had it up and running thanks to this guide.

Setup ngrok and run TensorBoard on Colab
https://www.dlology.com/blog/quick-guide-to-run-tensorboard-in-google-colab/

Hope it helps you!

[manivaradarajan]

The ngrok workflow is not officially supported.

Have you tried the TensorBoard notebook extension described here?

https://www.tensorflow.org/tensorboard/tensorboard_in_notebooks

[psyyip]

I had similar issue and it seemed that it had to do with ngrok that I had in assets folder being corrupted or had wrong permissions perhaps? Anyway, I got around it by downloading ngrok and i had it up and running thanks to this guide.

Setup ngrok and run TensorBoard on Colab
https://www.dlology.com/blog/quick-guide-to-run-tensorboard-in-google-colab/

Hope it helps you!

Sorry, I failed to tell you that I did run the following two commands beforehand.

! wget -c -nc https://bin.equinox.io/c/4VmDzA7iaHb/ngrok-stable-linux-amd64.zip ! unzip -o ngrok-stable-linux-amd64.zip

And below is the output after the installation.

`--2020-02-08 12:43:06-- https://bin.equinox.io/c/4VmDzA7iaHb/ngrok-stable-linux-amd64.zip
Resolving bin.equinox.io (bin.equinox.io)... 34.237.203.145, 34.206.126.139, 3.225.172.191, ...
Connecting to bin.equinox.io (bin.equinox.io)|34.237.203.145|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 13773305 (13M) [application/octet-stream]
Saving to: ‘ngrok-stable-linux-amd64.zip’

ngrok-stable-linux- 100%[===================>] 13.13M 18.5MB/s in 0.7s

2020-02-08 12:43:07 (18.5 MB/s) - ‘ngrok-stable-linux-amd64.zip’ saved [13773305/13773305]

Archive: ngrok-stable-linux-amd64.zip
inflating: ngrok
`

That means, I did install ngrok, but still had the errors I showed earlier.

Error in fact appeared after I run the following command:

!curl -s http://localhost:4040/api/tunnels | python3 -c \ "import sys, json; print(json.load(sys.stdin)['tunnels'][0]['public_url'])"

[psyyip]

The ngrok workflow is not officially supported.

Have you tried the TensorBoard notebook extension described here?

https://www.tensorflow.org/tensorboard/tensorboard_in_notebooks

Hi Mani, sorry, I'm not clear about your suggestion. Do you mean I should run the following commands in Jupyter Notebook?

%%bash docker run -it -p 8888:8888 -p 6006:6006 \
tensorflow/tensorflow:nightly-py3-jupyter

There are errors as follow:

`the input device is not a TTY

CalledProcessError Traceback (most recent call last)
in
----> 1 get_ipython().run_cell_magic('bash', '', 'docker run -it -p 8888:8888 -p 6006:6006 \\ntensorflow/tensorflow:nightly-py3-jupyter\n')

/usr/local/lib/python3.5/dist-packages/IPython/core/interactiveshell.py in run_cell_magic(self, magic_name, line, cell)
2357 with self.builtin_trap:
2358 args = (magic_arg_s, cell)
-> 2359 result = fn(args, *kwargs)
2360 return result
2361

/usr/local/lib/python3.5/dist-packages/IPython/core/magics/script.py in named_script_magic(line, cell)
140 else:
141 line = script
--> 142 return self.shebang(line, cell)
143
144 # write a basic docstring:

in shebang(self, line, cell)

/usr/local/lib/python3.5/dist-packages/IPython/core/magic.py in (f, a, *k)
185 # but it's overkill for just that one bit of state.
186 def magic_deco(arg):
--> 187 call = lambda f, a, *k: f(a, *k)
188
189 if callable(arg):

/usr/local/lib/python3.5/dist-packages/IPython/core/magics/script.py in shebang(self, line, cell)
243 sys.stderr.flush()
244 if args.raise_error and p.returncode!=0:
--> 245 raise CalledProcessError(p.returncode, cell, output=out, stderr=err)
246
247 def _run_script(self, p, cell, to_close):

CalledProcessError: Command 'b'docker run -it -p 8888:8888 -p 6006:6006 \\ntensorflow/tensorflow:nightly-py3-jupyter\n'' returned non-zero exit status 1`

Despite the error, I continue to run the following commands:

%load_ext tensorboard
%tensorboard --logdir OUTDIR

Then, another error:

4ac04ae6d17cd38-dot-us-west1.notebooks.googleusercontent.com took too long to respond.

Can you tell me what I did wrongly? What are the right steps? Thank you

[nickkimer]

The ngrok workflow is not officially supported.
Have you tried the TensorBoard notebook extension described here?
https://www.tensorflow.org/tensorboard/tensorboard_in_notebooks

Hi Mani, sorry, I'm not clear about your suggestion. Do you mean I should run the following commands in Jupyter Notebook?

%%bash docker run -it -p 8888:8888 -p 6006:6006 \
tensorflow/tensorflow:nightly-py3-jupyter

There are errors as follow:

`the input device is not a TTY

CalledProcessError Traceback (most recent call last)
in
----> 1 get_ipython().run_cell_magic('bash', '', 'docker run -it -p 8888:8888 -p 6006:6006 \ntensorflow/tensorflow:nightly-py3-jupyter\n')

/usr/local/lib/python3.5/dist-packages/IPython/core/interactiveshell.py in run_cell_magic(self, magic_name, line, cell)
2357 with self.builtin_trap:
2358 args = (magic_arg_s, cell)
-> 2359 result = fn(args, *kwargs)
2360 return result
2361

/usr/local/lib/python3.5/dist-packages/IPython/core/magics/script.py in named_script_magic(line, cell)
140 else:
141 line = script
--> 142 return self.shebang(line, cell)
143
144 # write a basic docstring:

in shebang(self, line, cell)

/usr/local/lib/python3.5/dist-packages/IPython/core/magic.py in (f, a, *k)
185 # but it's overkill for just that one bit of state.
186 def magic_deco(arg):
--> 187 call = lambda f, a, *k: f(a, *k)
188
189 if callable(arg):

/usr/local/lib/python3.5/dist-packages/IPython/core/magics/script.py in shebang(self, line, cell)
243 sys.stderr.flush()
244 if args.raise_error and p.returncode!=0:
--> 245 raise CalledProcessError(p.returncode, cell, output=out, stderr=err)
246
247 def _run_script(self, p, cell, to_close):

CalledProcessError: Command 'b'docker run -it -p 8888:8888 -p 6006:6006 \ntensorflow/tensorflow:nightly-py3-jupyter\n'' returned non-zero exit status 1`

Despite the error, I continue to run the following commands:

%load_ext tensorboard
%tensorboard --logdir OUTDIR

Then, another error:

4ac04ae6d17cd38-dot-us-west1.notebooks.googleusercontent.com took too long to respond.

Can you tell me what I did wrongly? What are the right steps? Thank you

I have the exact same errors..

[tarrade]

Same issue using GCP AI Platform notebook (TF 2.1 image)

I was told by the GCP AI Platform notebook team that this scenario is not supported because TensorBoard widget is trying to do port forwarding under the hood and the AI Platform notebook proxy prevents it from connecting to non-HTTPS ports.

1. Is there a tricks to run in another way to avoid this issue (even if TensorBoard in not running in a jupyter cell
2. Is there a plan to be able to have TensorBoard running on GCP AI Platform notebook ? I think this will be great for data scientist working on GCP and it is also use in GCP ML training

[nickkimer]

@tarrade thanks for your comment here. I have had to manually port forward while the model was training through SSH to my local and start tensorboard there instead. I can confirm that this method works as an alternative to using jupyter notebook on GCP

[tarrade]

@nickkimer what is your full solution to have it working on AI Platform notebook ? Thanks

[ghostandthemachine]

+1 on this. I get the same issue as @tarrade has shown above. I'm sort of getting around this by installing and using the jupyterlab-tensorboard extension (which required a pip install jupyter-tenorboard). This is less then ideal though and buggy. It's a holdover that might work temporarily for others while this gets sorted.

Would be ideal to have things "just work" like docs and examples as @tarrade shows above and also make sure the TB version is compatible with the TF version (which is not the case with the other extension).

Anyone have any updates on this getting fixed in GCP AI Hub notebooks?

[tarrade]

@ghostandthemachine @nickkimer I just opened a "feature request" https://github.com/tensorflow/tensorboard/issues/3408. Fee free to add your comments or requirements.

[psyyip]

I run the following in a Jupyter Notebook cell.

pip install jupyter-tensorboard
import tensorflow as tf
%load_ext tensorboard
%tensorboard --logdir logs

I still see the same error

4ac04ae6d17cd38-dot-us-west1.notebooks.googleusercontent.com took too long to respond.

Then, I run the following in another cell.

docker run -it -p 8888:8888 -p 6006:6006 tensorflow/tensorflow:nightly-py3-jupyter 

Another error appears

  File "<ipython-input-3-9b9451b6c838>", line 1
    docker run -it -p 8888:8888 -p 6006:6006 \
             ^
SyntaxError: invalid syntax

I add %%bash and run it again

%%bash
docker run -it -p 8888:8888 -p 6006:6006 \
tensorflow/tensorflow:nightly-py3-jupyter 

Error appears

the input device is not a TTY
---------------------------------------------------------------------------
CalledProcessError                        Traceback (most recent call last)
<ipython-input-4-b983203f88cf> in <module>
----> 1 get_ipython().run_cell_magic('bash', '', 'docker run -it -p 8888:8888 -p 6006:6006 \\\ntensorflow/tensorflow:nightly-py3-jupyter \n')

/usr/local/lib/python3.5/dist-packages/IPython/core/interactiveshell.py in run_cell_magic(self, magic_name, line, cell)
   2357             with self.builtin_trap:
   2358                 args = (magic_arg_s, cell)
-> 2359                 result = fn(*args, **kwargs)
   2360             return result
   2361 

/usr/local/lib/python3.5/dist-packages/IPython/core/magics/script.py in named_script_magic(line, cell)
    140             else:
    141                 line = script
--> 142             return self.shebang(line, cell)
    143 
    144         # write a basic docstring:

</usr/local/lib/python3.5/dist-packages/decorator.py:decorator-gen-110> in shebang(self, line, cell)

/usr/local/lib/python3.5/dist-packages/IPython/core/magic.py in <lambda>(f, *a, **k)
    185     # but it's overkill for just that one bit of state.
    186     def magic_deco(arg):
--> 187         call = lambda f, *a, **k: f(*a, **k)
    188 
    189         if callable(arg):

/usr/local/lib/python3.5/dist-packages/IPython/core/magics/script.py in shebang(self, line, cell)
    243             sys.stderr.flush()
    244         if args.raise_error and p.returncode!=0:
--> 245             raise CalledProcessError(p.returncode, cell, output=out, stderr=err)
    246 
    247     def _run_script(self, p, cell, to_close):

CalledProcessError: Command 'b'docker run -it -p 8888:8888 -p 6006:6006 \\\ntensorflow/tensorflow:nightly-py3-jupyter \n'' returned non-zero exit status 1

For me, pip install jupyter-tensorboard does not help. What did I miss?

[ghostandthemachine]

@psyyip I'm not running any docker so I can't help you there. From what you shared above though it looks like missed installing the jupyterlab-tensorboard extension. That can be install via the extensions manager (if you enable the extensions manager through the lab ui). Otherwise you can install it, check that projects README for the info on that.

Also, to be clear, this does not make the desired approach work in cell out form. This basically reverts to the previous approach which I think was based on the above mentioned plugin. To use it you will need to use the "create a tensorboard" command from the command pallet.

[psyyip]

@ghostandthemachine , can you tell me how to install the jupyterlab-tensorboard? I'm running notebook within GCP AI Platform. I run this command in a cell:

%%bash
jupyter labextension install jupyterlab_tensorboard

Then, error

An error occured.
PermissionError: [Errno 13] Permission denied: '/usr/local/share/jupyter/lab/extensions/jupyterlab_tensorboard-0.1.6.tgz'
See the log file for details:  /tmp/jupyterlab-debug-wswh5ql0.log

Within the Jupyter UI, I selected Settings-> Enable Extension Manager (Experimental). Then I run the above command again, still has the same error. Can you tell me the exact steps?

[ghostandthemachine]

For me I use pip install jupyter_tensorboard Then install jupyter lab tensorboard through the extension manager. Check the read me for that extension. I’ve never seen the permissions issue have installed this on a couple vms this past week.

[ghostandthemachine]

One other note, the magic tensorboard command still won’t work with the above. Per the docs, launch new tensorboards via the command pallet.

[sayakpaul]

Here are some suggestions:

• This process is a bit more involved. If you can write your event logs to a GCS Bucket and then trigger a TensorBoard instance from a Cloud Shell pointing towards the GCS Bucket that would work (you would need to change the ports for Web Preview).
• This one I tried with AWS EC2 but the same concept can be applied here as well. If you have SSH access to the instance then you can SSH into the instance with the particular board and launch your TensorBoard instance from there. The workflow for this one would be:

$ ssh -i ~/.ssh/id_rsa -L localhost:port_number:localhost:port_number username@ip_address

• After you are able to successfully log in you should be able to launch TensorBoard.

Hope this helps.

[ethem-kinginthenorth]

any update on this issue? still just workaround or if there is an official way of resolving this?

[bileschi]

I recognize that this is not suitable for many cases, and does not solve the root problem, but would using a hosted solution like TensorBoard.dev work for some people here?

[ethem-kinginthenorth]

Yes, I can use workaround including running everything on a local computer and that would not solve the root problem. Anyways. Thanks

[sayakpaul]

@ethem-kinginthenorth would this suggestion be more coherent with your use case?

[vlasenkoalexey]

I think the easiest way to workaround this is to run tensorboard in a JupyterLab console:
tensorboard --port=7000 --logdir logs

And then setup port forwarding using gcloud command:
gcloud beta compute ssh --zone "{your zone}" "{your instance name}" --project "{your project name}" -- -L 7000:localhost:7000

You can copy this command for your VM from gcloud console VM instance details page, click SSH drop down and select 'view gcloud command'. Then just add port you want to forward as -- -L 7000:localhost:7000

After this you should be able to access tensorboard running on your cloud VM as localhost:7000