Jupyterhub: Login -> Logout -> Login fails

[vbraun]

I can login once but not twice. Jupyterhub 0.5.0 running under a single user with pam authentication

$ ./jupyterhub --log-level=DEBUG
[D 2016-03-20 16:45:56.367 JupyterHub application:529] Looking for ./jupyterhub_config in None
[D 2016-03-20 16:45:56.367 JupyterHub application:549] Loaded config file: /home/vbraun-hub/JupyterHub/jupyterhub_config.py
[I 2016-03-20 16:45:56.369 JupyterHub app:558] Loading cookie_secret from /home/vbraun-hub/JupyterHub/jupyterhub_cookie_secret
[D 2016-03-20 16:45:56.369 JupyterHub app:623] Connecting to db: sqlite:///jupyterhub.sqlite
[W 2016-03-20 16:45:56.390 JupyterHub app:292] 
    Generating CONFIGPROXY_AUTH_TOKEN. Restarting the Hub will require restarting the proxy.
    Set CONFIGPROXY_AUTH_TOKEN env or JupyterHub.proxy_auth_token config to avoid this message.

[W 2016-03-20 16:45:56.393 JupyterHub app:685] No admin users, admin interface will be unavailable.
[W 2016-03-20 16:45:56.393 JupyterHub app:686] Add any administrative users to `c.Authenticator.admin_users` in config.
[I 2016-03-20 16:45:56.393 JupyterHub app:712] Not using whitelist. Any authenticated user will be allowed.
[D 2016-03-20 16:45:56.400 JupyterHub app:787] Loaded users: 
    vbraun-hub
[I 2016-03-20 16:45:56.405 JupyterHub app:1113] Hub API listening on http://127.0.0.1:8081/hub/
[I 2016-03-20 16:45:56.408 JupyterHub app:860] Starting proxy @ http://*:443/
[D 2016-03-20 16:45:56.408 JupyterHub app:861] Proxy cmd: ['configurable-http-proxy', '--ip', '', '--port', '443', '--api-ip', '127.0.0.1', '--api-port', '444', '--default-target', 'http://127.0.0.1:8081', '--ssl-key', 'letsencrypt/etc/live/vbraun.cc/privkey.pem', '--ssl-cert', 'letsencrypt/etc/live/vbraun.cc/cert.pem']
16:45:56.542 - info: [ConfigProxy] Proxying https://*:443 to http://127.0.0.1:8081
16:45:56.545 - info: [ConfigProxy] Proxy API at http://127.0.0.1:444/api/routes
[D 2016-03-20 16:45:56.612 JupyterHub app:889] Proxy started and appears to be up
[I 2016-03-20 16:45:56.612 JupyterHub app:1136] JupyterHub is now running at http://127.0.0.1:443/
[W 2016-03-20 16:46:03.348 JupyterHub auth:383] Failed to open PAM session for vbraun-hub: [PAM Error 14] Cannot make/remove an entry for the specified session
[I 2016-03-20 16:46:03.353 JupyterHub spawner:436] Spawning jupyterhub-singleuser --user=vbraun-hub --port=36298 --cookie-name=jupyter-hub-token-vbraun-hub --base-url=/user/vbraun-hub --hub-host= --hub-prefix=/hub/ --hub-api-url=http://127.0.0.1:8081/hub/api --ip=127.0.0.1
Failed to set groups [Errno 1] Operation not permitted
[D 2016-03-20 16:46:03.434 JupyterHub spawner:287] Polling subprocess every 30s
[W 2016-03-20 16:46:03.808 vbraun-hub notebookapp:253] ipywidgets package not installed.  Widgets are unavailable.
[I 2016-03-20 16:46:03.819 vbraun-hub notebookapp:1079] Serving notebooks from local directory: /home/vbraun-hub
[I 2016-03-20 16:46:03.819 vbraun-hub notebookapp:1079] 0 active kernels 
[I 2016-03-20 16:46:03.819 vbraun-hub notebookapp:1079] The Jupyter Notebook is running at: http://127.0.0.1:36298/user/vbraun-hub/
[I 2016-03-20 16:46:03.819 vbraun-hub notebookapp:1080] Use Control-C to stop this server and shut down all kernels (twice to skip confirmation).
[I 2016-03-20 16:46:03.826 vbraun-hub log:47] 302 GET /user/vbraun-hub (127.0.0.1) 0.58ms
[D 2016-03-20 16:46:03.827 JupyterHub utils:84] Server at http://127.0.0.1:36298/user/vbraun-hub responded with 302
[I 2016-03-20 16:46:03.827 JupyterHub base:301] User vbraun-hub server took 0.738 seconds to start
[I 2016-03-20 16:46:03.827 JupyterHub orm:159] Adding user vbraun-hub to proxy /user/vbraun-hub => http://127.0.0.1:36298
[D 2016-03-20 16:46:03.829 JupyterHub orm:146] Fetching POST http://127.0.0.1:444/api/routes/user/vbraun-hub
[D 2016-03-20 16:46:03.833 JupyterHub base:226] Setting cookie for vbraun-hub: jupyter-hub-token-vbraun-hub, {'secure': True}
[D 2016-03-20 16:46:03.835 JupyterHub base:226] Setting cookie for vbraun-hub: jupyter-hub-token, {'secure': True}
[I 2016-03-20 16:46:03.839 JupyterHub log:100] 302 POST /hub/login?next= (@::ffff:x.x.x.x) 791.45ms
[I 2016-03-20 16:46:03.839 JupyterHub login:79] User logged in: vbraun-hub
[D 2016-03-20 16:46:03.876 JupyterHub pages:29] User is running: /user/vbraun-hub
[I 2016-03-20 16:46:03.880 JupyterHub log:100] 302 GET /hub/ (vbraun-hub@::ffff:x.x.x.x) 3.17ms
[I 2016-03-20 16:46:03.914 vbraun-hub log:47] 302 GET /user/vbraun-hub (::ffff:x.x.x.x) 0.38ms
[I 2016-03-20 16:46:03.963 JupyterHub log:100] 200 GET /hub/api/authorizations/cookie/jupyter-hub-token-vbraun-hub/[secret] ([email protected]) 7.26ms
[I 2016-03-20 16:46:09.041 vbraun-hub log:47] 302 GET /user/vbraun-hub/logout (::ffff:x.x.x.x) 0.40ms
[I 2016-03-20 16:46:09.079 JupyterHub login:17] User logged out: vbraun-hub
[I 2016-03-20 16:46:09.084 JupyterHub log:100] 302 GET /hub/logout (vbraun-hub@::ffff:x.x.x.x) 5.37ms
[I 2016-03-20 16:46:09.118 JupyterHub log:100] 302 GET /hub/ (@::ffff:x.x.x.x) 0.59ms
[I 2016-03-20 16:46:09.174 JupyterHub log:100] 200 GET /hub/login (@::ffff:x.x.x.x) 20.75ms
[W 2016-03-20 16:46:13.159 JupyterHub auth:372] PAM Authentication failed (@::ffff:x.x.x.x): [PAM Error 7] Authentication failure
[D 2016-03-20 16:46:13.160 JupyterHub login:81] Failed login for None
[I 2016-03-20 16:46:13.164 JupyterHub log:100] 200 POST /hub/login?next= (@::ffff:x.x.x.x) 1586.23ms

The "Failed login for None" was already suspicious in #485

[minrk]

The "Failed login for None" is just a mis-typed log statement (fixed by #487). Are you certain the username+password are correct in the second login? Do you have any configuration when you see this?

[vbraun]

The username and password is saved in the browser and auto-filled in both attempts. I can't see the password but I definitely didn't enter anything different.

[minrk]

Does every subsequent login fail? Does it still fail if you manually type the password, discarding the autofill values?

[vbraun]

Yes. Digging deeper, this is denied by SELinux but only on the second login

type=AVC msg=audit(1458587705.198:25235): avc:  denied  { entrypoint } for  pid=30519 comm="jupyterhub" path="/usr/sbin/unix_chkpwd" dev="vda2" ino=413122 scontext=unconfined_u:unconfined_r:unconfined_t:s0 tcontext=system_u:object_r:chkpwd_exec_t:s0 tclass=file

It seems that you need MLS SystemHigh to call unix_chkpwd

# runcon unconfined_u:unconfined_r:unconfined_t:s0 /usr/sbin/unix_chkpwd
runcon: /usr/sbin/unix_chkpwd: Permission denied
# runcon unconfined_u:unconfined_r:unconfined_t:s0-s0:c0.c1023 /usr/sbin/unix_chkpwd
This binary is not designed for running in this way

Which the jupyterhub process has but jupyterhub-singleuser (which only exists after the first login) does not:

# ps auxwwZ | grep jupyterhub
unconfined_u:unconfined_r:unconfined_t:s0-s0:c0.c1023 vbraun-+ 3325 0.0  0.6 271580 36028 pts/2 S+ 21:29   0:00 /home/vbraun-hub/JupyterHub/tools/py3/bin/python /home/vbraun-hub/JupyterHub/tools/py3/bin/jupyterhub -f=./jupyterhub_config.py --log-level=DEBUG
unconfined_u:unconfined_r:unconfined_t:s0 vbraun-+ 3645 2.1  0.6 275328 39232 ? Ss  21:40   0:00 /home/vbraun-hub/JupyterHub/tools/py3/bin/python /home/vbraun-hub/JupyterHub/tools/py3/bin/jupyterhub-singleuser --user=vbraun-hub --port=59945 --cookie-name=jupyter-hub-token-vbraun-hub --base-url=/user/vbraun-hub --hub-host= --hub-prefix=/hub/ --hub-api-url=http://127.0.0.1:8081/hub/api --ip=127.0.0.1

[minrk]

I have no idea why SELinux would be denying every password check after the first.

[vbraun]

My guess is that this is a pamela bug; presumably there is something cached/left open so that subsequent logins misbehave. Similar PAM bugs can be found easily. Quoting https://bugs.freedesktop.org/show_bug.cgi?id=62866:

A PAM service needs to invoke the PAM session hooks like this:

pam_open_session(h);
pid = fork();
if (pid == 0) {
exec();
}
waitpid(pid);
pam_close_session(h);
exit();

That's the only correct way. i.e. the PAM handle can only be used once, and both pam_open_session() and pam_close_session() need to be called in the parent -- not the child. Also, since the session hooks will set all kinds of stuff like resource limits, security labels, audit info, selinux labels, yadda yadda yadda, the parent must exit() after the close session hook.

[minrk]

From the logs, it looks like it's not opening and then failing to close the session, it's failing to open the session even the first time. I guess that's leaving it in some dirty state maybe? Can you mock out pamela.(open|close)_session so it skips the session stuff, and see if that works? Do you have any more information about your system? I cannot reproduce anything like this, so I really have nothing to go on for debugging.

Re the process stuff, PAM calls are only ever made in the Hub parent process, never the jupyterhub-singleuser process, so I don't think that's it.

Since this seems to cause problems in different contexts, I'm currently inclined to add a flag for disabling the session open/close, and maybe even disable it by default. In most cases, it seems to either do nothing useful (this is the most common) or not work properly.

[vbraun]

For the record, this is on current CentOS (release 7.2.1511)

[vbraun]

When I just return in jupyterhub.auth.pre_spawn_start and post_spawn_stop then I can login/logout/login as I should. I'm a bit confused that the dummy pre_spawn_start is actually only called once and post_spawn_stop never, no matter how many times I login and logout. Though presumably jupyterhub expects that they return something?

The open_session indeed fails

In [1]: import pamela

In [2]: pamela.open_session('vbraun-hub', service='login')
---------------------------------------------------------------------------
PAMError                                  Traceback (most recent call last)
<ipython-input-2-ddb9c0bd5c04> in <module>()
----> 1 pamela.open_session('vbraun-hub', service='login')

/home/vbraun-hub/JupyterHub/tools/py3/lib/python3.5/site-packages/pamela.py in open_session(username, service, encoding)
    274 def open_session(username, service='login', encoding='utf-8'):
    275     handle = pam_start(service, username, encoding=encoding)
--> 276     return pam_end(handle, PAM_OPEN_SESSION(handle, 0))
    277 
    278 def close_session(username, service='login', encoding='utf-8'):

/home/vbraun-hub/JupyterHub/tools/py3/lib/python3.5/site-packages/pamela.py in pam_end(handle, retval)
    218     if retval == 0:
    219         retval = e
--> 220     raise PAMError(errno=retval)
    221 
    222 def authenticate(username, password=None, service='login', encoding='utf-8', resetcred=True):

PAMError: [PAM Error 14] Cannot make/remove an entry for the specified session

[minrk]

Sessions are opened/closed around Spawner start/stop, not login/logout, and logging out doesn't imply stopping the server. Maybe that's where it's going wrong. I'll do a bit of reading on PAM sessions. For now, there's #490 to just turn it off when it's not working right.

[vbraun]

You are not allowed to change /proc/self/loginuid once it is set; The syslog failure is

Mar 23 21:55:42 sage python[28558]: pam_loginuid(login:session): Cannot open /proc/self/loginuid: Permission denied
Mar 23 21:55:42 sage python[28558]: pam_loginuid(login:session): set_loginuid failed
Mar 23 21:55:42 sage python[28558]: pam_systemd(login:session): Failed to create session: Access denied

Pam-1.2.0 has a patch to not attempt to write if the loginuid is already the correct one (https://git.fedorahosted.org/cgit/linux-pam.git/commit/?id=45cdd2489e68465c2d2202370c350069d2a391b8) but CentOS 7 ships with pam-1.1.8.

I can call pamela.open_session successfully in my user account on Fedora 23 (pam-1.2.1).

[vbraun]

Still doesn't explain the bug here, catching the PAM error should be perfectly fine. We are already logged in under the correct uid after all.

The jupyterhub.auth.pre_spawn_start is called from the same pid as the jupyterhub process, and it seems that pamela never forks anything. According to Lennart Poettering (quoted above) that is incorrect; The process calling pam_open_session loses SElinux permissions which can never be re-gained (thats the whole point of SElinux). Instead, Jupyterhub should fork before calling pam_open_session and then waitpid and then pam_close_session in the child.

[minrk]

Just to be clear, that means for PAM sessions to work on 1.1, there need to be three processes:

1. Hub
2. fork that opens the session and does nothing else
3. the actual subprocess that uses the session

[vbraun]

Thats the way I understand it, but

• 2) can also fork further processes for that particular user
• This is not specific to any particular PAM version; On pam-1.1.8 the pam_open_session fails apparently further down in the chain of pam modules, after applying session restrictions. Even if it were to succeed, you'd have restricted the wrong (jupyterhub) process.

[minrk]

Hm, this is way more complicated than I thought, and a bad fit for JupyterHub. I now regret merging support for PAM sessions.

[minrk]

I also see no indication that the user processes must be subprocesses of the pam_open_session process, only time restriction. Is that true?

[vbraun]

You mean call pam_open_session / pam_close_session from the jupyter single user server? The obvious thing that would go wrong is that your notebook might die as you hack on C stuff, exceed memory limits, or whatever. Then you never call pam_close_session, utmp/wtmp is not updated, more depending on pam configuration (e.g. user home directory not unmounted)

Its quite easy to imagine stuff also requires a fork after pam_open_session, e.g. a SEliunx process transition. That of course would depend on the details of the pam setup. I haven't dug into the details but I'd expect breakage.

[minrk]

User code never runs in the server, so hacking on C code in the kernel would have a hard time killing the server, but it can be terminated uncleanly in other ways. What I actually meant to ask is if it would be okay to open/close the PAM sessions in processes that are _siblings_ of the single-user server, as opposed to parents. That would be easy to implement. I see basically no documentation about how PAM sessions affect the processes in which they are created or their subprocesses. All I can find is that "It should be possible for the PAM library to open a session and close the same session from different applications" which suggests (not conclusively) to me that it should be fine to open and close the session from independent processes.

[vbraun]

I don't see how thats possible, pam_open_session doesn't take a pid as argument. It applies to your own process only.

The way I interpret that cryptic "It should be possible for the PAM library to open a session and close the same session from different applications" is: You can pam_open_session, fork, and later call pam_close_session in the child. In any case its pretty clear that you need the pam handle to pass into pam_close_session, and there is no way to serialize it; Its an opaque pointer to some pam implementation details. So the only way to get it into a different process is via fork, and then only a child.

[willingc]

@minrk, @vbraun,

Closing this issue as it's been inactive for several months. Please feel free to open a new issue or reopen this issue. Thanks! :sunflower:

[natbusa]

Hi @minrk
I am building a vagrant centos 7 box (aiming for a later install on RHEL 7)
Python is built from source (Python 3.5.2) with openssl support

It's quite simple actually:
pamela does not work here:

[vagrant@datalab ~]$ python3 -m pamela -a amy
Password: 
[PAM Error 7] Authentication failure
[vagrant@datalab ~]$ 

unless you have sudo rights

[vagrant@datalab ~]$ sudo /usr/local/bin/python3 -m pamela -a usertest
Password: 
[vagrant@datalab ~]$ 

I am pretty sure this is the root cause for the thread above
Any idea how to debug/fix the pamela module?

[natbusa]

Solved:

it turns out that this had nothing to do with sudo, but more with the fact that the user jupyterhub had no access to the /etc/shadow file. As reported by @ybarraud https://medium.com/@ybarraud/setting-up-jupyterhub-with-sudospawner-and-anaconda-844628c0dbee#.vnol2aao4 in case of non-root users make sure that you have access to the file or otherwise PAM auth will fail.

One way of doing so if a system-wide sudo access (that was indeed why the above worked)
Another way suggested in the blog reported here is to add a group shadow and have /etc/shadow accessible by each user in the group shadow:

sudo groupadd shadow
sudo chgrp shadow /etc/shadow
sudo chmod g+r /etc/shadow
sudo usermod -a -G shadow jupyterhub

[vbraun]

@natbusa this is not the issue here.

Also, your solution significantly reduces the security of your system. Unless you are the only user, jupyterhub should most definitely not have read access to /etc/shadow.

[natbusa]

@vbraun thank you for your feedback. You don't need to take my word on it as this is currently the way non-root PAM auth is described on the jupyterhub pages. Please check it yourself https://github.com/jupyterhub/jupyterhub/wiki/Using-sudo-to-run-JupyterHub-without-root-privileges

If you have solved it in a better, more secure way, please do share your solution.

[vbraun]

Well there isn't a more secure way, thats what this ticket is about. Currently you just can't use jupyterhub local user authentication if you require state of the art security (i.e.: you can once, you just can't log in a second time).

[calz1]

@willingc I got this issue when trying out JupyterHub and it is quite confusing because PAM is the default setup. If this is still an outstanding problem, why close it?

[willingc]

@calz1 Which version of JupyterHub are you using? The original issue was for version 0.5 and the current stable is 0.7.2. If you are using the current stable, I would recommend opening a new issue with details on your configuration and debug logs. Thanks!

[jcanelhas]

I was having SELinux preventing access, the troubleshooter sugested these lines:
ausearch -c 'jupyterhub' --raw | audit2allow -M my-jupyterhub
semodule -i my-jupyterhub.pp

[jm2004]

@minrk, if I ma now using OAuth for handling login&logout, if I want to kill the container when I logout, is there anyway or some setting-up I can do to make it happen?
Thanks!

[jmchandonia]

I had this same bug with current stable. To fix it, I had to both set up the shadow group as described above, and also add this to my config file as described in #1138:

c.PAMAuthenticator.open_sessions = False

[akaszynski]

Had the same issue in the current stable release. Didn't have to setup the shadow group, just added the following to jupyterhub_config.py:

c.PAMAuthenticator.open_sessions = False

[brianwvincent]

I'n on Centos 7 and after initial login/logout, started experiencing PAM login issues. After making the following change to the config file, it started working as expected.

c.PAMAuthenticator.open_sessions = False