Notebook: Run jupyter notebook behind an NTLM corporate network

[dofine]

The network config in my company is with NTLM on Windows 7, causing really headache in python network communication.

It doesn't work when I run jupyter notebook and open localhost:port with my browser, responding Socket Error 10061. The IT guy doesn't seem to know to much so I wish I could get some help here.

After I config jupyter notebook to listen on all IP, I can open the homepage at http://myip:8888. However the kernel cannot be connected, even though the command line shows "Kernel started: xxxxx-xxxx-xxxx-xxxx".

[D 10:57:39.838 NotebookApp] Config changed:                                                                   
[D 10:57:39.838 NotebookApp] {'NotebookApp': {'ip': u'*', 'log_level': 10, 'open_browser': False}}             
[D 10:57:39.838 NotebookApp] Searching [u'C:\\Users\\n3132', 'C:\\Users\\n3132\\.jupyter', 'E:\\n3132\\Programs
\\anaconda\\etc\\jupyter', 'C:\\ProgramData\\jupyter'] for config files                                        
[D 10:57:39.838 NotebookApp] Looking for jupyter_config in C:\ProgramData\jupyter                              
[D 10:57:39.838 NotebookApp] Looking for jupyter_config in E:\n3132\Programs\anaconda\etc\jupyter              
[D 10:57:39.838 NotebookApp] Looking for jupyter_config in C:\Users\n3132\.jupyter                             
[D 10:57:39.838 NotebookApp] Looking for jupyter_config in C:\Users\n3132                                      
[D 10:57:39.838 NotebookApp] Looking for jupyter_notebook_config in C:\ProgramData\jupyter                     
[D 10:57:39.838 NotebookApp] Looking for jupyter_notebook_config in E:\n3132\Programs\anaconda\etc\jupyter     
[D 10:57:39.838 NotebookApp] Looking for jupyter_notebook_config in C:\Users\n3132\.jupyter                    
[D 10:57:39.838 NotebookApp] Looking for jupyter_notebook_config in C:\Users\n3132                             
[D 10:57:39.838 NotebookApp] Config changed:                                                                   
[D 10:57:39.848 NotebookApp] {'NotebookApp': {'log_level': 10, 'open_browser': False, 'ip': u'*'}}             
[W 10:57:40.279 NotebookApp] WARNING: The notebook server is listening on all IP addresses and not using encryp
tion. This is not recommended.                                                                                 
[W 10:57:40.279 NotebookApp] WARNING: The notebook server is listening on all IP addresses and not using authen
tication. This is highly insecure and not recommended.                                                         
[D 10:57:40.299 NotebookApp] Terminals not available (error was No module named fcntl)                         
[I 10:57:40.299 NotebookApp] Serving notebooks from local directory: C:\Users\n3132                            
[I 10:57:40.299 NotebookApp] 0 active kernels                                                                  
[I 10:57:40.299 NotebookApp] The Jupyter Notebook is running at: http://[all ip addresses on your system]:8888/

[I 10:57:40.299 NotebookApp] Use Control-C to stop this server and shut down all kernels (twice to skip confirm
ation).                    

[takluyver]

What browser are you using? It may not work on IE, especially if you're stuck on an older version.

Serving on all IP addresses without setting up a password is also a security risk, in case you weren't aware. I'm not sure about Windows network setups, but I would encourage you not to rely on that as a long term solution.

[dofine]

@takluyver Hi I've tried Chrome 49 and 50, neither works. I can open the homepage of the notebook, only cannot get the kernel connected. Maybe something todo with zeromq or websockets?

[takluyver]

I'd guess it's websockets that are the issue. Do you have any corporate internet security software? We've certainly had trouble with Sophos before, and I think some other security products can mess with websockets as well.

[dofine]

@takluyver We have McAfee here... Seems like it's not easy to make it work?

[takluyver]

I don't think McAfee is one we've had problems with.

Have you tried switching between localhost and 127.0.0.1? Sometimes one works but the other doesn't.

[dofine]

@takluyver Yes I've tried and only by setting ip to my actual internal ip address '10.255.224.54' can I open the notebook homepage. Then I see

'[D 22:38:43.332 NotebookApp] Connecting to: tcp://127.0.0.1:63843
[I 22:38:43.404 NotebookApp] Kernel started: 60803dd6-f282-461c-8b03-e329df8c3b09' 

in debug log.
Tried to edit jupyter_notebook_config.py with

# Set the kernel's IP address [default localhost]. If the IP address is
# something other than localhost, then Consoles on other machines will be able
# to connect to the Kernel, so be careful!
c.ConnectionFileMixin.ip = u'10.255.224.54'

But still no luck...

[willingc]

@takluyver What about trying 0.0.0.0 as the IP address in addition to localhost or 127.0.0.1?

[takluyver]

I think that IP address needs to be a real one, because the connection file is used to connect as well as to listen. 0.0.0.0 only works when you're listening, if I understand correctly.

[dofine]

If it helps, my network is forced to use a global proxy, so I cannot change proxy setup as the official docs say about proxy configuration. Also, in any Python scripts running on my PC, I've got to set os.environ["no_proxy"]=abc.com so that the scripts can get connected to abc.com. Or else Error 10061.

[takluyver]

Have you configured your browser to bypass the proxy for your own computer?

[willingc]

@takluyver I think it is worth giving 0.0.0.0 a shot if memory serves me I believe that we have done that with other notebook troubleshooting especially if there is a proxy or firewall. Run it in debug mode and look at the output.

[dofine]

I'm using Chrome and it's forced to use proxy, no means to bypass it...
Here are the relevant debug log. My pc ip address is 10.255.224.54

λ jupyter notebook --ip=0.0.0.0 --debug --no-browser
[D 15:17:38.148 NotebookApp] Searching [u'C:\\Users\\n3132', 'C:\\Users\\n3132\\.jupyter', 'E:\\n3132\\Programs\\anaconda2\\anaconda\\etc\\jupyter', 'C:\\ProgramData\\jupyter'] for config files
[D 15:17:38.148 NotebookApp] Looking for jupyter_config in C:\ProgramData\jupyter
[D 15:17:38.148 NotebookApp] Looking for jupyter_config in E:\n3132\Programs\anaconda2\anaconda\etc\jupyter
[D 15:17:38.148 NotebookApp] Looking for jupyter_config in C:\Users\n3132\.jupyter
[D 15:17:38.148 NotebookApp] Looking for jupyter_config in C:\Users\n3132
[D 15:17:38.148 NotebookApp] Looking for jupyter_notebook_config in C:\ProgramData\jupyter
[D 15:17:38.148 NotebookApp] Looking for jupyter_notebook_config in E:\n3132\Programs\anaconda2\anaconda\etc\jupyter
[D 15:17:38.148 NotebookApp] Looking for jupyter_notebook_config in C:\Users\n3132\.jupyter
[D 15:17:38.148 NotebookApp] Loaded config file: C:\Users\n3132\.jupyter\jupyter_notebook_config.py
[D 15:17:38.158 NotebookApp] Looking for jupyter_notebook_config in C:\Users\n3132
[D 15:17:38.812 NotebookApp] Terminals not available (error was No module named terminado)
[I 15:17:38.822 NotebookApp] Serving notebooks from local directory: C:\Users\n3132
[I 15:17:38.822 NotebookApp] 0 active kernels
[I 15:17:38.822 NotebookApp] The Jupyter Notebook is running at: http://0.0.0.0:8888/
[I 15:17:38.822 NotebookApp] Use Control-C to stop this server and shut down all kernels (twice to skip confirmation).
[D 15:17:43.609 NotebookApp] 200 GET /api/sessions?_=1461568544135 (192.168.72.213) 2.00ms
[D 15:17:43.878 NotebookApp] 200 GET /api/contents?type=directory&_=1461568544136 (192.168.72.213) 263.00ms
[D 15:17:45.789 NotebookApp] Using contents: services/contents
[D 15:17:45.842 NotebookApp] 200 GET /notebooks/Untitled.ipynb (192.168.72.213) 54.00ms
[D 15:17:45.855 NotebookApp] 304 GET /static/components/MathJax/MathJax.js?config=TeX-AMS_HTML-full,Safe&delayStartupUntil=configured (192.168.72.213) 1.00ms
[D 15:17:45.864 NotebookApp] 304 GET /custom/custom.css (192.168.72.213) 1.00ms
[D 15:17:45.880 NotebookApp] 304 GET /static/components/MathJax/config/TeX-AMS_HTML-full.js?rev=2.5.3 (192.168.72.213) 1.00ms
[D 15:17:45.938 NotebookApp] 304 GET /static/components/MathJax/config/Safe.js?rev=2.5.3 (192.168.72.213) 0.00ms
[D 15:17:46.148 NotebookApp] 200 GET /static/services/contents.js?v=20160425151738 (192.168.72.213) 1.00ms
[D 15:17:46.160 NotebookApp] 200 GET /custom/custom.js?v=20160425151738 (192.168.72.213) 1.00ms
[D 15:17:46.176 NotebookApp] 200 GET /api/config/common?_=1461568666027 (192.168.72.213) 1.00ms
[D 15:17:46.177 NotebookApp] 200 GET /api/config/notebook?_=1461568666026 (192.168.72.213) 1.00ms
[D 15:17:46.239 NotebookApp] Native kernel (python2) available from E:\n3132\Programs\anaconda2\anaconda\lib\site-packages\ipykernel\resources
[D 15:17:46.250 NotebookApp] Native kernel (python2) available from E:\n3132\Programs\anaconda2\anaconda\lib\site-packages\ipykernel\resources
[D 15:17:46.253 NotebookApp] 304 GET /api/kernelspecs (192.168.72.213) 18.00ms
[D 15:17:46.282 NotebookApp] 304 GET /static/components/MathJax/jax/output/HTML-CSS/fonts/STIX-Web/fontdata.js?rev=2.5.3 (192.168.72.213) 1.00ms
[D 15:17:46.296 NotebookApp] 304 GET /static/components/MathJax/extensions/Safe.js?rev=2.5.3 (192.168.72.213) 1.00ms
[D 15:17:47.002 NotebookApp] 200 GET /api/contents/Untitled.ipynb?type=notebook&_=1461568666028 (192.168.72.213) 704.00ms
[D 15:17:47.012 NotebookApp] 200 GET /nbextensions/widgets/notebook/js/extension.js?v=20160425151738 (192.168.72.213) 10.00ms
[D 15:17:47.058 NotebookApp] No kernel name specified, using default kernel
[D 15:17:47.144 NotebookApp] Native kernel (python2) available from E:\n3132\Programs\anaconda2\anaconda\lib\site-packages\ipykernel\resources
[D 15:17:47.144 NotebookApp] Starting kernel: ['E:\\n3132\\Programs\\anaconda2\\anaconda\\python.exe', '-m', 'ipykernel', '-f', u'C:\\Users\\n3132\\AppData\\Roaming\\jupyter\\runtime\\kernel-0acc5887-2bd6-40de-b3d1-371fe6496768.json']
[D 15:17:47.154 NotebookApp] Connecting to: tcp://10.255.224.54:56799
[I 15:17:47.161 NotebookApp] Kernel started: 0acc5887-2bd6-40de-b3d1-371fe6496768
[D 15:17:47.164 NotebookApp] Kernel args: {'kernel_name': None, 'cwd': u'C:\\Users\\n3132'}
[D 15:17:47.164 NotebookApp] 201 POST /api/sessions (192.168.72.213) 107.00ms
[D 15:17:47.193 NotebookApp] 200 GET /api/contents/Untitled.ipynb/checkpoints?_=1461568666029 (192.168.72.213) 4.00ms
[D 15:17:47.255 NotebookApp] 200 GET /nbextensions/widgets/notebook/js/widgetarea.js?v=20160425151738 (192.168.72.213) 17.00ms
[D 15:17:47.272 NotebookApp] 200 GET /nbextensions/widgets/widgets/js/init.js?v=20160425151738 (192.168.72.213) 4.00ms
[D 15:17:47.344 NotebookApp] 200 GET /nbextensions/widgets/widgets/js/manager.js?v=20160425151738 (192.168.72.213) 2.00ms
[D 15:17:47.352 NotebookApp] 200 GET /nbextensions/widgets/widgets/js/widget_link.js?v=20160425151738 (192.168.72.213) 5.00ms
[D 15:17:47.375 NotebookApp] 200 GET /nbextensions/widgets/widgets/js/widget_button.js?v=20160425151738 (192.168.72.213) 10.00ms
[D 15:17:47.375 NotebookApp] 200 GET /nbextensions/widgets/widgets/js/widget_box.js?v=20160425151738 (192.168.72.213) 0.00ms
[D 15:17:47.385 NotebookApp] 200 GET /nbextensions/widgets/widgets/js/widget_bool.js?v=20160425151738 (192.168.72.213) 0.00ms
[D 15:17:47.387 NotebookApp] 200 GET /nbextensions/widgets/widgets/js/widget_image.js?v=20160425151738 (192.168.72.213) 0.00ms
[D 15:17:47.387 NotebookApp] 200 GET /nbextensions/widgets/widgets/js/widget_int.js?v=20160425151738 (192.168.72.213) 0.00ms
[D 15:17:47.387 NotebookApp] 200 GET /nbextensions/widgets/widgets/js/widget.js?v=20160425151738 (192.168.72.213) 0.00ms
[D 15:17:47.387 NotebookApp] 200 GET /nbextensions/widgets/widgets/js/widget_float.js?v=20160425151738 (192.168.72.213) 0.00ms
[D 15:17:47.390 NotebookApp] 200 GET /nbextensions/widgets/widgets/js/widget_color.js?v=20160425151738 (192.168.72.213) 1.00ms
[D 15:17:47.421 NotebookApp] 200 GET /nbextensions/widgets/widgets/js/widget_output.js?v=20160425151738 (192.168.72.213) 0.00ms
[D 15:17:47.441 NotebookApp] 200 GET /nbextensions/widgets/widgets/js/widget_selection.js?v=20160425151738 (192.168.72.213) 20.00ms
[D 15:17:47.441 NotebookApp] 200 GET /nbextensions/widgets/widgets/js/widget_string.js?v=20160425151738 (192.168.72.213) 0.00ms
[D 15:17:47.461 NotebookApp] 200 GET /nbextensions/widgets/widgets/js/widget_selectioncontainer.js?v=20160425151738 (192.168.72.213) 0.00ms
[D 15:17:47.461 NotebookApp] 200 GET /nbextensions/widgets/widgets/js/widget_controller.js?v=20160425151738 (192.168.72.213) 0.00ms
[D 15:17:47.473 NotebookApp] 200 GET /static/components/backbone/backbone-min.js?v=20160425151738 (192.168.72.213) 0.00ms
[D 15:17:47.509 NotebookApp] 304 GET /nbextensions/widgets/widgets/css/widgets.min.css (192.168.72.213) 1.00ms

[willingc]

@dofine The detailed debug output is helpful.

Let's see if @takluyver can determine the root cause with this additional information. Here's what I'm seeing in the above debug output:

• no terminado installed
• 192.168.72.213 for nb server
• 10.255.224.54 for pc local machine
• looks like there is a mismatch between C: and E: drive usage

Starting kernel: ['E:\\n3132\\Programs\\anaconda2\\anaconda\\python.exe', '-m', 'ipykernel', '-f', u'C:\\Users\\n3132\\AppData\\Roaming\\jupyter\\runtime\\kernel-0acc5887-2bd6-40de-b3d1-371fe6496768.json']

@takluyver Thoughts? Install/upgrade IPython? Terminado? Review config file settings?

[dofine]

@willingc Wow, the nb server address is 192.168.72.213? I missed that. That looks like our company proxy server... Open Chrome Dev Tools and I see Remote Address is also 192.168.72.213. But really I'm trying to run nb server on my own working PC...

[takluyver]

I think 192.168.72.213 in those logs indicates where the HTTP requests are coming from, so it makes sense that that's the proxy server. I'd guess that the proxy is not smart enough to handle websockets.

@dofine The default when you set up a proxy should be to bypass it for localhost (aka 127.0.0.1), because there's no way that can work through the proxy. E.g. here's the proxy configuration dialog in Firefox - those hosts are excepted by default:

IIRC, Chrome uses the system proxy config, whereas Firefox has its own settings, so you may be able to configure this in Firefox even if you're locked out of it on Chrome. If you really can't configure it to bypass the proxy at all, even for localhost, and if the proxy can't handle websockets, then you're kind of stuck... the only thing I can think of is to run the notebook in a virtual machine where you would be able to configure proxy exclusions.

[dofine]

@takluyver Really great and thanks so much!

[takluyver]

Great, did you get it working?

[dofine]

Really sorry for the late reply.
Using firefox works.
Go to hell Chrome

[pbdanny]

Hi, I am also experienced same problems, running local Jupyter notebook with Google Chrome. The work around I use

1. Disconnecting from corporate network (disable WiFi or unplug LAN)
2. Connect Jupyter Notebook.
3. Reconnect with corporate network, the connection with Jupyter server still persist until disconnect from Jupyter kernel.

Preferred this methods, 'cos Jupyter font in Firefox looks wired.

[dhirschfeld]

IIUC chrome gets it's proxy settings from Internet Explorer so if you correctly configure localhost bypass in IE it should also work in Chrome.