Notebook: Interpretation of console output

[silhouetted]

Hi,

System + setup issue

I am running a Jupyter notebook on a virtual private server (VPS) Ubuntu 18.04.

The dockerfile is Jupyter Notebook Deep Learning Stack.

The OS version:

cat /proc/version
Linux version 4.15.0-62-generic (buildd@lcy01-amd64-024) (gcc version 7.4.0 (Ubuntu 7.4.0-1ubuntu1~18.04.1)) #69-Ubuntu SMP Wed Sep 4 20:55:53 UTC 2019

I am not an expert in Linux (although I like it), or networking, so I have tried to take reasonable security precautions:

1. I connect to my VPS through SSH on port 22. It uses public/private encryption key, rather than a password.
2. I start the notebook dockerfile in a terminal from SSH, which generates an encryption key, rather than a password (which is a different key each time the command is run).
3. I access the notebook through the web, through port 8888. This is over HTTP rather than HTTPS (should I worry about this?).

The issue

When I leave the notebook running, I get lots of attempted connections in the terminal log, none of which are from me:

[W 23:45:14.145 NotebookApp] 405 CONNECT check.best-proxies.ru:80 (95.213.187.186) 2.09ms referer=None
[I 02:28:27.195 NotebookApp] 302 GET / (177.54.62.194) 1.31ms
[I 03:18:18.132 NotebookApp] 302 GET / (118.47.72.153) 1.67ms
[W 03:45:37.157 NotebookApp] 403 POST http://check.best-proxies.ru/azenv.php?s=156895113717804PC238846785208888 (95.213.187.190): '_xsrf' argument missing from POST
[W 03:45:37.158 NotebookApp] 403 POST http://check.best-proxies.ru/azenv.php?s=156895113717804PC238846785208888 (95.213.187.190) 1.82ms referer=https://best-proxies.ru/
[I 03:45:37.159 NotebookApp] Malformed HTTP message from 95.213.187.190: Malformed HTTP request line
[W 03:45:42.330 NotebookApp] 405 CONNECT check.best-proxies.ru:80 (95.213.187.187) 2.28ms referer=None
[I 06:29:38.905 NotebookApp] 302 GET / (66.130.214.128) 0.85ms
[I 06:44:54.086 NotebookApp] 302 GET / (71.6.146.185) 0.95ms
[I 06:44:54.802 NotebookApp] 302 GET /tree (71.6.146.185) 1.45ms
[W 07:39:24.302 NotebookApp] 403 POST http://check.best-proxies.ru/azenv.php?s=156896516435819PC238846785208888 (95.213.187.186): '_xsrf' argument missing from POST
[W 07:39:24.305 NotebookApp] 403 POST http://check.best-proxies.ru/azenv.php?s=156896516435819PC238846785208888 (95.213.187.186) 4.45ms referer=https://best-proxies.ru/
[I 07:39:24.306 NotebookApp] Malformed HTTP message from 95.213.187.186: Malformed HTTP request line
[W 07:39:29.102 NotebookApp] 405 CONNECT check.best-proxies.ru:80 (95.213.187.188) 1.42ms referer=None

I looked at the Jupyter notebook documentation and searched the web to interpret these logs.

Request ports and methods

GET, POST and CONNECT appear to be HTTP request methods. The requests either state port 80 or no port.

I have run from another device: sudo nmap -sT -p- <my.vps.ip.address>.

Which informs me that the only open ports are the desired ones (22 SSH, 8787 rstudio, 8888 jupyter).

Request result

The three digit codes look like HTTP status codes (e.g. 403). I searched the Jupyter documentation for each code, and it only seems to explicitly refer to 403:

"By default, requests get a 403 forbidden response if the ‘Host’ header shows that the browser thinks it’s on a non-local domain. Setting this option to True disables this check."

Codes 405 and 302 are not mentioned, although in general as HTTP codes they appear to refer to "Method not allowed" and "Moved temporarily", respectively.

My interpretation of all this

So, after all this, I am inclined to interpret the results as:

1. A random internet port scanner found port an open port.
2. A number of connection attempts were made.
3. These attempts were unsuccessful. There is nothing to worry about and no need to change any security settings.

Would this be a reasonable interpretation of these logs?

[Jifan-Zhang]

I'm suffering from exactly the same matter.

[silhouetted]

Yes - I am pretty sure it is just random port scanning and if there was a successful connection I'd see more output, like a new notebook being created or a kernel starting.

The problem is that I am not 100% sure.

[Jifan-Zhang]

Yeah, I have more than 30 port-scanning logs and failure connection logs every day. Surprisingly, these connecting IPs are totally different.

[silhouetted]

FWIW I also posted about this issue here: https://discourse.jupyter.org/t/interpretation-of-console-output/2194

But I also did not get any responses! So I am a bit stuck - let me know if you get anywhere...

[bollwyvl]

Yes, pretty much any public-facing server will be port scanned constantly.

The codes are part of the HTTP standard (we didn't make them up). Here are
the meanings of the response codes:

https://developer.mozilla.org/en-US/docs/Web/HTTP/Status

Or more officially:

https://www.w3.org/Protocols/rfc2616/rfc2616-sec10.html

It's _probably_ okay, but if you start installing third-party
serverextensions you haven't audited, protecting the notebook server is
important. When running servers, defense in depth is your friend, and
reducing the number of open ports, and using non-standard ports, is better
than hoping everything is secure.

If you do not wish to see these log messages, and since you already have
ssh, you may wish to use a tunnel.

https://www.digitalocean.com/community/tutorials/how-to-route-web-traffic-securely-without-a-vpn-using-a-socks-tunnel

As it mentions, a VPN would be the gold standard, but it's more complex.

On Tue, Oct 8, 2019, 05:02 silhouetted [e-mail] wrote:

FWIW I also posted about this issue here:
https://discourse.jupyter.org/t/interpretation-of-console-output/2194

But I also did not get any responses! So I am a bit stuck - let me know if
you get anywhere...

—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub
https://github.com/jupyter/notebook/issues/4903?email_source=notifications&email_token=AAALCRGD24IVZX6JREZNTGLQNRECXA5CNFSM4IYUP6NKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEATOZVI#issuecomment-539421909,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AAALCRFUB3E4Z5HMNR2CG5DQNRECXANCNFSM4IYUP6NA
.

[silhouetted]

Thanks a lot for this reply - really reassuring, as I am not using any serverextensions. Will look into using an SSH tunnel.

Strangely, there is also another output message I am seeing a lot of now:

zmq message arrived on closed channel

I can see from the documentation that zmq is the protocol that the kernel uses to interface with the front-end.

The odd thing is that this message never appeared before and now appears to be there whenever I am running a notebook. Is there any reason why this might be?