Jupyterhub: PAM authentication errors

[robdempsey]

Hi

My setup

Python 3.4.3 |Continuum Analytics, Inc.| (default, Jun 4 2015, 15:29:08)
[GCC 4.4.7 20120313 (Red Hat 4.4.7-1)] on linux

jupyterhub        - JupyterHub: A multi-user server for Jupyter notebooks
  INSTALLED: 0.3.0.dev0
  LATEST:    0.2.0

For PAM authentication if I am using user accounts on the system is there anything to setup? I am getting the following errors?

[I 2015-10-15 10:19:18.788 JupyterHub app:1054] JupyterHub is now running at http://10.24.130.109:9999/
[I 2015-10-15 10:19:26.987 JupyterHub log:100] 302 GET / (@172.25.196.43) 2.09ms
[I 2015-10-15 10:19:27.033 JupyterHub log:100] 302 GET /hub (@172.25.196.43) 0.48ms
[I 2015-10-15 10:19:27.080 JupyterHub log:100] 302 GET /hub/ (@172.25.196.43) 1.04ms
[I 2015-10-15 10:19:27.157 JupyterHub log:100] 200 GET /hub/login (@172.25.196.43) 31.25ms
[W 2015-10-15 10:19:33.664 JupyterHub auth:215] PAM Authentication failed: [PAM Error 14] Cannot make/remove an entry for the specified session
[D 2015-10-15 10:19:33.666 JupyterHub login:81] Failed login for None
[I 2015-10-15 10:19:33.673 JupyterHub log:100] 200 POST /hub/login?next= (@172.25.196.43) 41.51ms

[D 2015-10-15 10:20:00.520 JupyterHub app:817] Proxy started and appears to be up
[I 2015-10-15 10:20:00.521 JupyterHub app:1054] JupyterHub is now running at http://10.24.130.109:9999/
[W 2015-10-15 10:20:08.519 JupyterHub auth:215] PAM Authentication failed: [PAM Error 7] Authentication failure
[D 2015-10-15 10:20:08.521 JupyterHub login:81] Failed login for None
[I 2015-10-15 10:20:08.561 JupyterHub log:100] 200 POST /hub/login?next= (@172.25.196.43) 1669.00ms

Thanks

Rob

[minrk]

Are you running JupyterHub as root?

[robdempsey]

Hi

No. I configured it using

https://github.com/jupyter/jupyterhub/wiki/Using-sudo-to-run-JupyterHub-without-root-privileges

Rob

[colbrydi]

Rob,

Did you make any progress on this problem?

I am new to jupyterhub but I did manage to spin up a VM today with a similar configuration to yours. I came across this post because I am also getting related errors.

I am running a CentOS7 VM with anaconda3 running jupyterhub configured using the same link you provided.

I have three test users and am able to log in with any of them. However, if I try to log in a second user or try to logout and back in I get the following error:

[W 2015-10-29 01:19:47.129 JupyterHub auth:215] PAM Authentication failed: [PAM Error 14] Cannot make/remove an entry for the specified session

My hope is to get this working for a intro to programming class we are going to hold early next year. Sorry I don't have a solution yet, but I will let you know if I learn anything.

• Dirk

[robdempsey]

Hi

No not at the moment. I have had to park this due to other priorities. Any help would be much appreciated

The progress I made was that

• I have never had to configure PAM before however I know the machines I work on the sshd turns it off

sed -i -e 's/UsePAM yes/UsePAM no/' /etc/ssh/sshd_config

not sure how relevant that is

• A web page I found

https://wiki.ceh.ac.uk/download/attachments/248089368/Tessella%20-%20JupyterHub%20details.txt?version=1&modificationDate=1436451901000&api=v2

has some PAM tweaks

• I had to install a new module pamela. Don't know if the problem resides there or not

Rob

[colbrydi]

Rob,

I got mine working this morning. For me there was an issue in how PAM was set up. Specifically the pam_loginuid.so option. I commended out the following line in the /etc/pam.d/login file:

session required pam_loginuid.so

Although I am still not sure what this change is doing, it fixed my immediate problem. I still need to verify (for my own peace of mind) that this change is not causing some other type of security error.

Here are some links to websites I used that helped me come up with this potential solution:

http://linux.die.net/man/8/pam_loginuid
https://forums.opensuse.org/showthread.php/462193-cron-and-PAM-ERROR-(Cannot-make-remove-an-entry-for-the-specified-session)
https://bugs.centos.org/view.php?id=2191
https://www.centos.org/forums/viewtopic.php?t=29905

Hope this helps,

• Dirk

[chengts95]

I have PAM error when using jupyterhub 0.3.0,too.
I update jupyterhub from 0.2.0 to 0.3.0 via pip.
It's running on CentOS 7,python 3.5.
At first, you can login normally. But after the first login, all the login attempts will fail and get a [PAM Error 7].
It doesn't happen in jupyterhub 0.2.0, In 0.2.0 all things are working well.

[robdempsey]

Hi

Dirk's comment above fixed the issue

Edit the file

/etc/pam.d/login

and put a comment infront of the pam_loginuid.so

session optional pam_loginuid.so

When I had a chance to read up on PAM pam_loginuid.so is related with auditing (I am at a POC stage so not needed)

Rob

[chengts95]

Sadly, it doesn't work for me :(.
In my /etc/pam.d/login pam_loginuid.so is not optional, and the comment doesn't work for me.
I think the problem is caused by jupyterhub itself.

[marcjoos]

Hi,

I had the exact same issue (with JupyterHub 0.3.0, Python 3.5.1 on a RHEL 7 machine). I can have one successful login and then all following attempts fail. Commenting the pam_loginuid.so line in /etc/pam.d/login didn't help. I tried to comment the pam_selinux.so lines and then it worked... Kinda black magic for me, but I'm used to have issues due to SELinux!

[minrk]

I suspect the later failures is due to failing to properly close the PAM session. I think I need to read up on exactly what should be done there, and at least make the behavior optional, so you can turn it off if it's not quite right.

[aubreybailey]

Confirmed this to be related to selinux.
I don't recommend doing this, but:
'setenforce 0'
Brand new install on Centos 7 restores to full functionality with no other changes except this one.

[chicagobuss]

'setenforce 0' basically turns off selinux entirely.. that's not really a good idea!

In my setup I was trying to run the jupyterhub server as root with sudo -u jupyterhub <cmd>. This was so I didn't have to give the jupyterhub user a login shell.

I got some help making a proper selinux module.. here's what it looks like if you want to do this instead.

Put this in sudo_exec_selinux.te:

module sudo_exec 1.1;

require {
        type unconfined_t;
        type sudo_exec_t;
        class file { read entrypoint };
}

#============= unconfined_t ==============
allow unconfined_t sudo_exec_t:file entrypoint;

Make a module with it:
checkmodule -M -m -o sudo_exec_selinux.mod sudo_exec_selinux.te
Make a package with the module:
semodule_package -o sudo_exec_selinux.pp -m sudo_exec_selinux.mod
Install the package:
semodule -i sudo_exec_selinux.pp

[chicagobuss]

...and if you want to run jupyterhub as a service (as you probably will), you'll need several policies. I ended up needing all three of these.. if some SELinux guru could help explain/clean this up a bit I'd certainly appreciate it!

Put this in sudo_exec_selinux.te:

module sudo_exec 1.1;

require {
        type unconfined_t;
        type sudo_exec_t;
        class file { read entrypoint };
}

#============= unconfined_t ==============
allow unconfined_t sudo_exec_t:file entrypoint;

Make a module with it:
checkmodule -M -m -o sudo_exec_selinux.mod sudo_exec_selinux.te

Make a package with the module:
semodule_package -o sudo_exec_selinux.pp -m sudo_exec_selinux.mod

Install the package:
semodule -i sudo_exec_selinux.pp

Put this in service_sudo_exec.te:

module service_sudo_exec 1.1;

require {
        type unconfined_service_t;
        type unconfined_t;
        class process { transition };
}

#============= unconfined_service_t ==============
allow unconfined_service_t unconfined_t:process transition;

Make a module with it:
checkmodule -M -m -o service_sudo_exec.mod service_sudo_exec.te

Make a package with the module:
semodule_package -o service_sudo_exec.pp -m service_sudo_exec.mod

Install the package:
semodule -i service_sudo_exec.pp

Put this in chkpwd_exec.te:

module chkpwd_exec 1.1;

require {
        type unconfined_t;
        type chkpwd_exec_t;
        class file{ entrypoint };
}

#============= unconfined_t ==============
allow unconfined_t chkpwd_exec_t:file entrypoint;

Make a module with it:
checkmodule -M -m -o chkpwd_exec.mod chkpwd_exec.te

Make a package with the module:
semodule_package -o chkpwd_exec.pp -m chkpwd_exec.mod

Install the package:
semodule -i chkpwd_exec.pp

[janmtl]

+1 I'm getting the exact same problem as described here. Tried commenting out everything in the /etc/pam.d/login file and saw no effects.

I note that I also get the following error when running this command as my _adminuser_

> python -m pamela -a 'user1'
Password:
[PAM Error 7] Authentication failure

but running the same command as _user1_ itself produces no error.

[sirgogo]

same boat here with latest minoconda3 docker.

made an admin account with:
useradd admin -p pass123 -g sudo

but get authentication error when I try to login via web interface:
[W 2016-07-21 04:17:34.808 JupyterHub auth:386] PAM Authentication failed (admin@::ffff:172.17.0.1): [PAM Error 7] Authentication failure

running as root, since inside a docker container. There really should be a better way, besides disabling PAM!?

[angelapper]

same problem, when I want to setup jupyterhub on centos, did we get a solution?

[sirgogo]

I gave up, switched to Intel Python which has a conda install that works. Alternatively, you might try adduser instead of useradd, since I've read there are some subtleties.

[younseunghyun]

there is no any progress for this issue? I encountered this issue with Centos7.

Jupyterhub emit error PAM Error 7 when second user trying to login.

[younseunghyun]

Hi, I solve this issue by setting this

c.PAMAuthenticator.open_sessions = False

My version is 0.6.1

I can't expect what kind of issue will occurred by this setting.

I will post here when I encounter other problem.

[willingc]

The current stable version is JupyterHub 0.7.2. I'm going to close this issue. Please feel free to open a new issue if you see similar behavior with release 0.7.2. Thank you.

[MDMBD]

I work on 4 VM centos 7 locally (http):
configuration:

• jupyterhub --version
  0.7.2
• python --version
  Python 3.5.2 :: Anaconda 4.1.1 (64-bit)

[I 2017-03-29 19:06:08.184 JupyterHub app:1485] JupyterHub is now running at http://localhost:8000/
[I 2017-03-29 19:06:54.084 JupyterHub log:100] 302 GET / (@10.16.79.166) 4.89ms
[I 2017-03-29 19:06:54.101 JupyterHub log:100] 302 GET /hub (@10.16.79.166) 1.15ms
[I 2017-03-29 19:06:54.117 JupyterHub log:100] 302 GET /hub/ (@10.16.79.166) 2.81ms
[I 2017-03-29 19:06:54.126 JupyterHub log:100] 302 GET /login (@10.16.79.166) 2.38ms
[I 2017-03-29 19:06:54.232 JupyterHub log:100] 200 GET /hub/login (@10.16.79.166) 98.42ms
[W 2017-03-29 19:07:31.118 JupyterHub auth:471] PAM Authentication failed ([email protected]): [PAM Error 7] Authentication failure
[I 2017-03-29 19:07:31.131 JupyterHub log:100] 200 POST /hub/login?next= (@10.16.79.166) 2127.06ms

I corrected this:

#session required pam_loginuid.so

and

 c.PAMAuthenticator.open_sessions = False 

and this

#%PAM-1.0
auth [user_unknown=ignore success=ok ignore=ignore default=bad] pam_securetty.so
auth       substack     system-auth
auth       include      postlogin
account    required     pam_nologin.so
account    include      system-auth
password   include      system-auth
# pam_selinux.so close should be the first session rule
session    required     pam_selinux.so close
# session    required     pam_loginuid.so
session    optional     pam_console.so
# pam_selinux.so open should only be followed by sessions to be executed in the user context
# session    required     pam_selinux.so open
session    required     pam_namespace.so
session    optional     pam_keyinit.so force revoke
session    include      system-auth
session    include      postlogin
-session   optional     pam_ck_connector.so

Thank you for your help

[willingc]

@MDMBD It would help me a bit if you could clarify your question. Thanks!

[MDMBD]

HI,

When I connect to the jupyterhub interface via a localhost: 8000 URL.
I receive an error message on my vm cento7.
Here's the message:

1-If user no admin I receive this message:

[I 2017-03-30 09:53:35.951 JupyterHub app:1453] Hub API listening on http://localhost:54321/hub/
[W 2017-03-30 09:53:35.960 JupyterHub app:1174] Running JupyterHub without SSL.  I hope there is SSL termination happening somewhere else... 

And In jupyterhub interface I receive this messge

[I 2017-03-30 09:53:35.960 JupyterHub app:1176] Starting proxy @ http:// localhost:8000/
09:53:36.458 - info: [ConfigProxy] Proxying http://localhost:8000 to http:// localhost:54321
09:53:36.471 - info: [ConfigProxy] Proxy API at http:// localhost:5432/api/routes
[I 2017-03-30 09:53:36.579 JupyterHub app:1485] JupyterHub is now running at http:// localhost:8000/
[I 2017-03-30 09:54:54.509 JupyterHub spawner:783] Spawning jupyterhub-singleuser '--user="team_k"' '--cookie-name="jupyter-hub-token-team_k"' '--base-url="/user/team_kleber"' '--hub-host=""' '--hub-prefix="/hub/"' '--hub-api-url="http:// localhost:54321/hub/api"' '--ip="127.0.0.1"' --port=37186
[E 2017-03-30 09:54:54.518 JupyterHub spawner:793] Permission denied trying to run '/root/anaconda3/bin/jupyterhub-singleuser'. Does team_k have access to this file?
[E 2017-03-30 09:54:54.527 JupyterHub user:251] Unhandled error starting team_k's server: [Errno 13] Permission denied
[E 2017-03-30 09:54:54.578 JupyterHub web:1548] Uncaught exception POST /hub/login?next= (10.16.79.166)
    HTTPServerRequest(protocol='http', host='localhost:8000', method='POST', uri='/hub/login?next=', version='HTTP/1.1', remote_ip='10.16.79.166', headers={'Accept': 'application/x-ms-application, image/jpeg, application/xaml+xml, image/gif, image/pjpeg, application/x-ms-xbap, */*', 'Cache-Control': 'no-cache', 'Content-Length': '36', 'X-Forwarded-Proto': 'http', 'Accept-Language': 'fr-FR', 'X-Forwarded-Port': '8000', 'Content-Type': 'application/x-www-form-urlencoded', 'Accept-Encoding': 'gzip, deflate', 'Connection': 'close', 'X-Forwarded-For': '10.16.79.166', 'Dnt': '1', 'User-Agent': 'Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 6.1; WOW64; Trident/7.0; SLCC2; .NET CLR 2.0.50727; .NET CLR 3.5.30729; .NET CLR 3.0.30729; Media Center PC 6.0; .NET CLR 1.1.4322; .NET4.0C; .NET4.0E; InfoPath.3)', 'X-Forwarded-Host': ' localhost:8000', 'Host': localhost:8000', 'Referer': 'http:// localhost:8000/hub/login'})
    Traceback (most recent call last):
      File "/root/anaconda3/lib/python3.5/site-packages/tornado/web.py", line 1469, in _execute
        result = yield result
      File "/root/anaconda3/lib/python3.5/site-packages/jupyterhub/handlers/login.py", line 84, in post
        yield self.spawn_single_user(user)
      File "/root/anaconda3/lib/python3.5/site-packages/jupyterhub/handlers/base.py", line 328, in spawn_single_user
        yield gen.with_timeout(timedelta(seconds=self.slow_spawn_timeout), f)
      File "/root/anaconda3/lib/python3.5/site-packages/jupyterhub/user.py", line 261, in spawn
        raise e
      File "/root/anaconda3/lib/python3.5/site-packages/jupyterhub/user.py", line 229, in spawn
        ip_port = yield gen.with_timeout(timedelta(seconds=spawner.start_timeout), f)
      File "/root/anaconda3/lib/python3.5/types.py", line 243, in wrapped
        coro = func(*args, **kwargs)
      File "/root/anaconda3/lib/python3.5/site-packages/jupyterhub/spawner.py", line 787, in start
        start_new_session=True, # don't forward signals
      File "/root/anaconda3/lib/python3.5/subprocess.py", line 947, in __init__
        restore_signals, start_new_session)
      File "/root/anaconda3/lib/python3.5/subprocess.py", line 1551, in _execute_child
        raise child_exception_type(errno_num, err_msg)
    PermissionError: [Errno 13] Permission denied
[E 2017-03-30 09:54:54.697 JupyterHub log:99] {
      "Accept": "application/x-ms-application, image/jpeg, application/xaml+xml, image/gif, image/pjpeg, application/x-ms-xbap, */*",
      "Cache-Control": "no-cache",
      "Content-Length": "36",
      "X-Forwarded-Proto": "http",
      "Accept-Language": "fr-FR",
      "X-Forwarded-Port": "8000",
      "Content-Type": "application/x-www-form-urlencoded",
      "Accept-Encoding": "gzip, deflate",
      "Connection": "close",
      "X-Forwarded-For": "10.16.79.166",
      "Dnt": "1",
      "User-Agent": "Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 6.1; WOW64; Trident/7.0; SLCC2; .NET CLR 2.0.50727; .NET CLR 3.5.30729; .NET CLR 3.0.30729; Media Center PC 6.0; .NET CLR 1.1.4322; .NET4.0C; .NET4.0E; InfoPath.3)",
      "X-Forwarded-Host": " localhost:8000",
      "Host": " localhost:8000",
      "Referer": "http://localhost:8000/hub/login"
    }
[E 2017-03-30 09:54:54.697 JupyterHub log:100] 500 POST /hub/login?next= (@10.16.79.166) 302.29ms

And
In jupyterhub interface I receive this messge

500 : Internal Server Error 
Failed to start your server. Please contact admin. 

2-If user admin I receive this message:

[E 2017-03-30 10:03:45.035 JupyterHub log:100] 500 POST /hub/login?next= (@10.16.79.166) 167.53ms
[I 2017-03-30 10:03:58.426 JupyterHub log:100] 302 GET / (@10.16.79.166) 2.05ms
[I 2017-03-30 10:03:58.428 JupyterHub log:100] 302 GET /hub (@10.16.79.166) 0.63ms
[I 2017-03-30 10:03:58.435 JupyterHub log:100] 302 GET /hub/ (@10.16.79.166) 1.37ms
[I 2017-03-30 10:03:58.441 JupyterHub log:100] 302 GET /login (@10.16.79.166) 1.29ms
[I 2017-03-30 10:03:58.460 JupyterHub log:100] 200 GET /hub/login (@10.16.79.166) 15.11ms
[W 2017-03-30 10:04:35.886 JupyterHub auth:471] PAM Authentication failed ([email protected]): [PAM Error 7] Authentication failure
[I 2017-03-30 10:04:35.899 JupyterHub log:100] 200 POST /hub/login?next= (@10.16.79.166) 1655.52ms

And
In jupyterhub interface I receive this messge

Sign in 
Warning: JupyterHub seems to be served over an unsecured HTTP connection. We strongly recommend enabling HTTPS for JupyterHub. 
Invalid username or password 
Username: Password

[venkatchetkuri]

I have installed JupyterHub, and i can run the server and can login using a user called "livy".
I'm facing issue while trying to login with a different user:

"[W 2017-06-20 23:29:47.305 JupyterHub auth:471] PAM Authentication failed (user1@::ffff:98.190.141.61): [PAM Error 7] Authentication failure"

I've attached configuration file for reference.

Can anyone help me fix this issue ?

$ jupyterhub --no-ssl
[I 2017-06-20 23:29:27.673 JupyterHub app:724] Loading cookie_secret from /home/livy/jupyterhub_cookie_secret
[W 2017-06-20 23:29:27.708 JupyterHub app:365]
Generating CONFIGPROXY_AUTH_TOKEN. Restarting the Hub will require restarting the proxy.
Set CONFIGPROXY_AUTH_TOKEN env or JupyterHub.proxy_auth_token config to avoid this message.

[I 2017-06-20 23:29:27.739 JupyterHub app:1453] Hub API listening on http://127.0.0.1:8081/hub/
[W 2017-06-20 23:29:27.743 JupyterHub app:1174] Running JupyterHub without SSL. I hope there is SSL termination happening somewhere else...
[I 2017-06-20 23:29:27.743 JupyterHub app:1176] Starting proxy @ http://:8000/
23:29:27.886 - info: [ConfigProxy] Proxying http://:8000 to http://127.0.0.1:8081
23:29:27.890 - info: [ConfigProxy] Proxy API at http://127.0.0.1:8001/api/routes
[I 2017-06-20 23:29:27.948 JupyterHub app:1485] JupyterHub is now running at http://127.0.0.1:8000/
[I 2017-06-20 23:29:33.805 JupyterHub log:100] 200 GET /hub/login?next= (@::ffff:98.190.141.61) 32.59ms
[W 2017-06-20 23:29:47.305 JupyterHub auth:471] PAM Authentication failed (user1@::ffff:98.190.141.61): [PAM Error 7] Authentication failure
[I 2017-06-20 23:29:47.311 JupyterHub log:100] 200 POST /hub/login?next= (@::ffff:98.190.141.61) 1980.86ms
Configuration file for jupyterhub.txt

[goforthanddie]

Actually I have the same issue. System: CentOS 7.3, jupyterhub 0.8.1 in an anaconda 3 5.0.1 environment with python 3.6. I can login with one local user, but when I logout and try to log back into the hub or a second user, it gives:

[W 2018-01-23 13:23:27.276 JupyterHub auth:563] PAM Authentication failed (dll01@myip): [PAM Error 7] Authentication failure
[W 2018-01-23 13:23:27.277 JupyterHub base:351] Failed login for dll01

secure log file reads:

Jan 23 13:24:14 localhost python: pam_unix(login:auth): authentication failure; logname=myusername uid=0 euid=0 tty= ruser= rhost=  user=dll01

I figured, that only when I shutdown jupyterhub the session gets closed, it stays open even if I use the logout button:

Jan 23 13:24:57 localhost python: pam_unix(login:session): session closed for user dll01

So maybe that's the issue. When I manipulate /etc/pam.d/login and remove

session    required     pam_selinux.so open

it works fine. But actually I don't think that's the solution. Sooo... any one knows what's going on here? I know too little about PAM and sessions, but it seems to me that there is a mistake.

Best
Samy

[dturaev]

I believe this is related to https://github.com/jupyterhub/jupyterhub/issues/278, I observed the same effect (apart from internal server error 500). Seems that this is a selinux thing, disabling selinux or setting the c.pamauthenticator.open_sessions attribute should help.