Captum: Captum Insights not working in SageMaker

[austinmw]

When I try to run Captum Insights from a SageMaker notebook terminal on port 6006 by browsing to <sagemaker_notebook_address>/proxy/6006/, the tab name shows "Captum Insights", but the web page is blank. The same method works fine on my local system, or fine with tensorboard/flask apps through SageMaker. It seems to be a problem with Captum+SageMaker specifically.

Alternatively, when attempting to run tutorials/CIFAR_TorchVision_Captum_Insights.ipynb I get this error from within a notebook:

(I get the same error with visualizer.render(), just with less details)

---

Details:

I upgraded my SageMaker pytorch_p36 conda environment to torch==1.3.0. I installed captum from source with git clone https://github.com/pytorch/captum.git and then installed Insights with:

conda install -c conda-forge yarn
BUILD_INSIGHTS=1 python setup.py develop

Then ran the example with python captum/insights/example.py

And tried to access via <sagemaker_notebook_address>/proxy/6006/ (the same way I access a running tensorboard server)

I also tried it with/without modifying line 66 in insights/server.py from tcp.bind(("", 0)) to tcp.bind(("", 6006)) in order to use port 6006 (since this port seemed to work fine for running a tensorboard server).

[orionr]

Thanks, @austinmw, for debugging. I'll need to see if we can get access to a SageMaker instance. @edward-io or @J0Nreynolds, any thoughts here?

[neelamgehlot]

This could be related to https://github.com/awslabs/amazon-sagemaker-examples/issues/908 .

[neelamgehlot]

Hi @orionr,

This issue is reproducible with local jupyter installation as well, you won't need SageMaker instance.

Steps to reproduce

1. Install Jupyter (I installed anaconda that has jupyter as one of it's package bundle)
2. Install and enable Jupyter proxy extension conda install -c conda-forge nbserverproxy && jupyter serverextension enable --py nbserverproxy --sys-prefix
3. Create pytorch environment conda create -n pytorch_env -c pytorch pytorch torchvision
4. Activate the environment source activate pytorch_env
5. Install Captum Insights conda install -c pytorch pytorch torchvision
6. Run Captum Insights example python -m captum.insights.example
7. Run Jupyter server in another terminal jupyter notebook
8. Access Captum Insights via Jupyter proxy http://localhost:8888/proxy/<Captum Insights port>

[edward-io]

Thanks for the repro steps @neelamgehlot. When you run step 6, its launched as a separate web server that is not tied to Jupyter at all. I tested your steps, and they seem to work on the latest master, which will be released soon as 0.2.0. We have build instructions on how to build master on the README, please let me know that works for you.

I also spent some time setting up an AWS account for SageMaker and attempting to build from master from the notebook, but failed due to some strange interaction between SageMaker and Yarn. I'll try again soon.

[neelamgehlot]

Hi @edward-io,

I had installed Captum from conda https://anaconda.org/pytorch/captum which is at v0.1.0, I haven't tried with v0.2.0.

When you run step 6, its launched as a separate web server that is not tied to Jupyter at all.

That's right, with Jupyter nbserverproxy extension installed in step 2, you can access any other process via Jupyter using /proxy/<PORT> (see step 8) .

You can read more about nbserverproxy extension here https://github.com/jupyterhub/jupyter-server-proxy .

[edward-io]

@neelamgehlot I got Insights to work locally, but SageMaker has a bug in the proxy: https://forums.aws.amazon.com/message.jspa?messageID=914445#914445. Is this something you could look into on your side?

[NarineK]

@austinmw, @edward-io , can this issue be closed ?

[edward-io]

Closing issue as this is an issue with SageMaker, not Captum.