Tensorboard: Running Tensorboard 1.15 behind proxy - Projector

[tanguycdls]

Environment information (required)

Diagnostics

Diagnostics output

``````
--- check: autoidentify
INFO: diagnose_tensorboard.py version 393931f9685bd7e0f3898d7dcdf28819fef54c43

--- check: general
INFO: sys.version_info: sys.version_info(major=3, minor=6, micro=8, releaselevel='final', serial=0)
INFO: os.name: posix
INFO: os.uname(): posix.uname_result(sysname='Linux', nodename='tcd-0', release='4.14.104-95.84.amzn2.x86_64', version='#1SMP Sat Mar 2 00:40:20 UTC 2019', machine='x86_64')
INFO: sys.getwindowsversion(): N/A

--- check: package_management
INFO: has conda-meta: True
INFO: $VIRTUAL_ENV: None

--- check: installed_packages
INFO: installed: tb-nightly==1.15.0a20190723
WARNING: no installation among: ['tensorflow', 'tensorflow-gpu', 'tf-nightly', 'tf-nightly-2.0-preview', 'tf-nightly-gpu','tf-nightly-gpu-2.0-preview']
WARNING: no installation among: ['tensorflow-estimator', 'tensorflow-estimator-2.0-preview', 'tf-estimator-nightly']

--- check: tensorboard_python_version
INFO: tensorboard.version.VERSION: '1.15.0a20190723'

--- check: tensorflow_python_version
Traceback (most recent call last):
File "diagnose_tensorboard.py", line 419, in main
suggestions.extend(check())
File "diagnose_tensorboard.py", line 77, in wrapper
result = fn()
File "diagnose_tensorboard.py", line 236, in tensorflow_python_version
import tensorflow as tf
ModuleNotFoundError: No module named 'tensorflow'

--- check: tensorboard_binary_path
INFO: which tensorboard: b'/opt/conda/envs/moksha/bin/tensorboard\n'

--- check: readable_fqdn
INFO: socket.getfqdn(): 'tcd-0'

--- check: stat_tensorboardinfo
INFO: directory: /tmp/.tensorboard-info
INFO: os.stat(...): os.stat_result(st_mode=16895, st_ino=113246326, st_dev=96, st_nlink=2, st_uid=0, st_gid=0, st_size=28,st_atime=1563953558, st_mtime=1563960989, st_ctime=1563960989)
INFO: mode: 0o40777

--- check: source_trees_without_genfiles
INFO: tensorboard_roots (1): ['/opt/conda/envs/moksha/lib/python3.6/site-packages']; bad_roots (0): []

--- check: full_pip_freeze
INFO: pip freeze --all:
absl-py==0.7.1
aiohttp==3.5.4
asn1crypto==0.24.0
async-timeout==3.0.1
attrs==19.1.0
backcall==0.1.0
bleach==3.1.0
boto3==1.9.162
botocore==1.12.163
certifi==2019.6.16
cffi==1.11.5
chardet==3.0.4
Click==7.0
cloudpickle==1.2.1
cryptography==2.7
cycler==0.10.0
Cython==0.29.6
cytoolz==0.9.0.1
daal4py==2019.4
dask==2.1.0
dask-glm==0.2.0
decorator==4.4.0
defusedxml==0.6.0
docutils==0.14
entrypoints==0.3
future==0.17.1
grpcio==1.22.0
HeapDict==1.0.0
idna==2.6
idna-ssl==1.1.0
ipykernel==5.1.1
ipython==7.6.1
ipython-genutils==0.2.0
ipywidgets==7.5.0
jedi==0.14.1
Jinja2==2.10.1
jmespath==0.9.4
joblib==0.13.2
jsonschema==3.0.1
jupyter==1.0.0
jupyter-client==5.3.1
jupyter-console==6.0.0
jupyter-core==4.5.0
jupyter-server-proxy==1.1.0
karma-perf==114
kiwisolver==1.0.1
llvmlite==0.29.0
Markdown==3.1.1
MarkupSafe==1.1.1
matplotlib==3.0.3
mistune==0.8.4
mkl-fft==1.0.13
mkl-random==1.0.4
msgpack==0.6.1
multidict==4.5.2
multipledispatch==0.6.0
nb-conda-kernels==2.2.2
nbconvert==5.5.0
nbformat==4.4.0
notebook==6.0.0
numba==0.45.0
numexpr==2.6.8
numpy==1.16.2
pandas==0.24.1+0.g1700680.dirty
pandocfilters==1.4.2
parso==0.5.1
pexpect==4.7.0
pickleshare==0.7.5
Pillow==6.1.0
pip==10.0.1
prometheus-client==0.7.1
prompt-toolkit==2.0.9
protobuf==3.9.0
ptyprocess==0.6.0
pyarrow==0.11.1
pycparser==2.18
Pygments==2.4.2
pyOpenSSL==17.5.0
pyparsing==2.2.0
pyrsistent==0.15.3
PySocks==1.6.7
python-dateutil==2.6.0
pytz==2018.4
PyYAML==5.1.1
pyzmq==18.0.2
qtconsole==4.5.1
s3transfer==0.2.0
scikit-learn==0.20.3
scipy==1.2.0
Send2Trash==1.5.0
setuptools==41.0.1
simpervisor==0.3
six==1.11.0
sklearn==0.0
sortedcontainers==2.1.0
tb-nightly==1.15.0a20190723
TBB==0.1
terminado==0.8.2
testpath==0.4.2
toolz==0.10.0
torch==1.1.0
tornado==6.0.3
traitlets==4.3.2
typing-extensions==3.7.4
urllib3==1.24.1
wcwidth==0.1.7
webencodings==0.5.1
Werkzeug==0.15.5
wheel==0.31.0
widgetsnbextension==3.5.0
yarl==1.3.0
zict==1.0.0

``````

Next steps

No action items identified. Please copy ALL of the above output,
including the lines containing only backticks, into your GitHub issue
or comment. Be sure to redact any sensitive information.

Issue description

Issue exists only on nightly version ! 1.14 OK.

I'm running the Tensorboard version nightly 1.15 behind a jupyter-server-proxy in a notebook Kubeflow cluster. I use Jupyter server proxy to reverse proxy Tensorboard from port 6006 to the address notebooks/kubeflow/{kubeflowNotebookPath}/proxy/6006/.

• I run tensorboard without specifying any prefix path.

• Scalars / histograms and Distributions are working great : js files / images points out directly to /kubeflow/{kubeflowNotebookPath}/proxy/6006/ without any additional config.

• However projector does not work correctly, it just shows a white frame. I checked the developper debugger and it seems Tensorboard tries to load directly an index.js file without adding the proxy path. (https://kubeflow-ui-.*.*/data/plugin/projector/index.js)

• I did not find any other tabs of Tensorboard with similar issues and everything else is working great 👍

On my side, I installed back 1.14 and everything worked again,

Best

[wchargin]

Hi @tanguycdls! Thanks for the clear report. This is likely due to
changes in #2356. We’ll take a look.

[mengxr]

@wchargin It seems the issue came from https://github.com/tensorflow/tensorboard/blob/1.15/tensorboard/components/tf_tensorboard/tf-tensorboard.html#L905.

In our setup, TensorBoard is behind a port-forwarding proxy at https://hostname/proxy. This anonymous function becomes

~javascript
setTimeout(() => {
const base = document.createElement("base");
base.setAttribute("href", "https://hostname/proxy/data/plugin/projector/");
document.head.appendChild(base);
import("/data/plugin/projector/index.js").then((m) => void m.render());
}, 0);
~

The import("/data/plugin/projector/index.js") will try to load https://hostname/data/plugin/projector/index.js which does not exist.

Changing import to import("./index.js") would work for my case but I don't know if it is generally correct.

[mengxr]

@tanguycdls As a workaround before we have a proper fix, you can visit https://hostname/proxy/data/plugin/projector/projector_binary.html directly to see the projector page.

[mengxr]

@wchargin Is reverse proxy a scenario TensorBoard supports? It is a quite common setup.

[tanguycdls]

@mengxr Thanks for the tip on /plugin/projector/projector_binary.html I'm running into the same issue with the profiler. Any idea what could be the url ? I found:
proxy/6006/data/plugin/profile/trace_viewer_index.html but it does not work because trace url is not provided.

Thanks

[adampl]

@mengxr Yes, it's supported, check the --path_prefix option. However, apparently some plugins don't respect this option at the moment (like the profile plugin in my case).

[adampl]

In case of the Profile plugin, it's already fixed but not released yet - but a nightly build package should already support it, see: https://github.com/tensorflow/profiler/issues/45