Packer: ssh: Using multiple AuthenticationMethods

[easyas314159]

I'm not sure if this counts as a bug or a feature request. I have run into a situation where we have an AMI that has DUO installed (configured using the default setup here: https://duo.com/docs/duounix) to provide two-factor auth when you connect to the machine. We are using packer 1.0.0 to build a new AMI based on the image with DUO installed. The default user account (in our case ubuntu) has been configured to allow 2FA to be bypassed so packer can provision the instance. However, we keep getting the following error:

Error waiting for SSH: ssh: handshake failed: ssh: unable to authenticate, attempted methods [none publickey], no supported methods remain

With PACKER_LOG=1 and -debug I see the following:

2017/05/04 17:24:59 packer: 2017/05/04 17:24:59 [INFO] Attempting SSH connection...
2017/05/04 17:24:59 packer: 2017/05/04 17:24:59 reconnecting to TCP connection for SSH
2017/05/04 17:24:59 packer: 2017/05/04 17:24:59 handshaking with SSH
2017/05/04 17:25:00 packer: 2017/05/04 17:25:00 handshake error: ssh: handshake failed: ssh: unable to authenticate, attempted methods [none publickey], no supported methods remain
2017/05/04 17:25:00 packer: 2017/05/04 17:25:00 [DEBUG] SSH handshake err: ssh: handshake failed: ssh: unable to authenticate, attempted methods [none publickey], no supported methods remain
2017/05/04 17:25:00 packer: 2017/05/04 17:25:00 [DEBUG] Detected authentication error. Increasing handshake attempts.

When packer tries to connect I see this in /var/log/auth.log in the target machine:

May  4 23:14:20 ip-172-31-23-63 sshd[1911]: Did not receive identification string from X.X.X.X
May  4 23:14:20 ip-172-31-23-63 sshd[1912]: Connection closed by X.X.X.X port 59394 [preauth]

Using OpenSSH from the command-line works fine and I have included the relevant portion of ssh's verbose output below.

debug1: Authentications that can continue: publickey
debug3: start over, passed a different list publickey
debug3: preferred publickey,keyboard-interactive,password
debug3: authmethod_lookup publickey
debug3: remaining preferred: keyboard-interactive,password
debug3: authmethod_is_enabled publickey
debug1: Next authentication method: publickey
debug1: Offering RSA public key: /Users/kloney/.ssh/kevinl-veerum.rsa
debug3: send_pubkey_test
debug2: we sent a publickey packet, wait for reply
debug1: Authentications that can continue: publickey
debug1: Offering RSA public key: /Users/kloney/.ssh/id_rsa
debug3: send_pubkey_test
debug2: we sent a publickey packet, wait for reply
debug1: Authentications that can continue: publickey
debug1: Trying private key: ec2_ubuntu-xenial-16.04-amd64-server.pem
debug1: read PEM private key done: type RSA
debug3: sign_and_send_pubkey: RSA aa:4e:80:60:ea:d8:fa:d1:13:59:ee:3c:9e:d2:e9:ee
debug2: we sent a publickey packet, wait for reply
Authenticated with partial success.
debug1: Authentications that can continue: keyboard-interactive
debug3: start over, passed a different list keyboard-interactive
debug3: preferred publickey,keyboard-interactive,password
debug3: authmethod_lookup keyboard-interactive
debug3: remaining preferred: password
debug3: authmethod_is_enabled keyboard-interactive
debug1: Next authentication method: keyboard-interactive
debug2: userauth_kbdint
debug2: we sent a keyboard-interactive packet, wait for reply
debug2: input_userauth_info_req
debug2: input_userauth_info_req: num_prompts 0
debug3: packet_send2: adding 48 (len 6 padlen 10 extra_pad 64)
debug1: Authentication succeeded (keyboard-interactive).

I see the following when OpenSSH connects in /var/log/auth.log on the target machine:

May  4 23:13:39 ip-172-31-23-63 sshd[1842]: Accepted keyboard-interactive/pam for ubuntu from X.X.X.X port 59218 ssh2
May  4 23:13:39 ip-172-31-23-63 sshd[1842]: pam_unix(sshd:session): session opened for user ubuntu by (uid=0)
May  4 23:13:39 ip-172-31-23-63 systemd-logind[1116]: New session 7 of user ubuntu.

My guess is packer is having trouble handling the request for zero pieces of keyboard authentication information.

[mwhooker]

we don't do any keyboard-interactive authentication unless we're trying to type in a password.

It would probably not be too difficult to add, but it's not something we're focusing on now. See https://github.com/hashicorp/packer/blob/20b53f5ca3a02b2e087e4e34699790120e537571/builder/amazon/common/ssh.go#L103

You might be able to add a simple auth method which just hits enter or even asks the user to enter the MFA key. If you end up making that patch, it would probably have to live behind a flag

[easyas314159]

After taking a look at the code I would argue that the SSH config implementation is actually incorrect. Ignoring my MFA case, it is not uncommon to configure sshd to require both a certificate and the user's password which the existing code would also choke on.

I tested this yesterday by replicating the KeyboardInteractive from the password only case into the certificate only case of the SSH code and resolved the issue I was experiencing:

Instead of constructing 3 different ssh.ClientConfig objects depending on the scenario a case could be made for instead constructing a list of acceptable auth methods that always includes Password and KeyboardInteractive.

I'll put together a pull request for early next week.

[rickard-von-essen]

Could you provide the content of /etc/pam.conf and /etc/pam.d/ (with any secrets stripped).

[easyas314159]

/etc/pam.conf is empty

/etc/pam.d/sshd

#@include common-auth
auth  [success=1 default=ignore] /lib64/security/pam_duo.so
auth  requisite pam_deny.so
auth  required pam_permit.so

/etc/pam.d/common-auth

auth            requisite       pam_unix.so nullok_secure
auth            [success=1 default=ignore]      /lib64/security/pam_duo.so 
auth            requisite       pam_deny.so 
auth            required        pam_permit.so 

Nothing else in any other pam files has been added or modified

[mwhooker]

packer is designed to run unattended, so we don't expect to see a case where the remote end is requiring additional information. We can support interactive authentication, but it can't change existing behavior.

We can add a flag to the ssh config which would, as a special case, allow for keyboard-interactive authentication, but if that flag is not set, the build should fail as it does now.

Happy to work more with you on a PR

[rickard-von-essen]

@easyas314159 I think you are doing it a bit wrong. Instead of allowing ubuntu login without 2FA you should allow ssh key login which is (generally) stronger than 2FA.

Properly handling MFA for SSH would require a bigger overhaul of how we do keyboard interactive authentication and I for most it's better to use ssh keys instead.

[rickard-von-essen]

But If you don't allow using ssh keys you can modify pam.d/sshd to something like this (untested from the top of my head):

auth  [success=2 default=ignore] pam_succeed_if.so quiet user = ubuntu
auth  [success=1 default=ignore] /lib64/security/pam_duo.so
auth  requisite pam_deny.so
auth  required pam_permit.so

This would make the ubuntu user skip the DUO pam module. (Be sure to verify that this is correct before releasing this to PROD 😱).

See http://www.linux-pam.org/Linux-PAM-html/sag-pam_succeed_if.html

[easyas314159]

@rickard-von-essen You misunderstand, our environment requires both a SSH key, and a 2FA token, we still require the SSH key for the ubuntu user but allow the ubuntu user to skip the 2FA token when provisioning. You can see this in the verbose dump from ssh in my initial post.

[easyas314159]

I am not sure I have been clear enough as to my use case here, and I think there is a misunderstanding in packer's SSH code as to how SSH authentication actually works.

Let's ignore my MFA/2FA case here for a moment.

As of right now if you have a server that requires both an SSH key, and a password (e.g. passing ssh_password and ssh_agent_auth) then SSHConfig will return a ssh.ClientConfig that is only configured to support ssh-agent based authentication.

When packer attempts to establish an SSH connection, the server will reply with a request for a public key which will succeed and you'll get a partial success. The problem occurs when the SSH server says "ok great your key worked but let us start this all over again but I need a password this time". Packer halts and fails to authenticate because there are no more authentication methods available in the ssh.ClientConfig.

SSHConfig will only ever return one of the following mutually exclusive ssh.ClientConfigs:

1. Use the SSH agent
2. Use a provided private key
3. Use keyboard interactive or password with the value supplied in ssh_password

To properly interact with SSH SSHConfig should be returning a _single_ ssh.ClientConfig with one or more of the following ssh.AuthMethods:

• The SSH agent if it is running
• The private key file if it has been supplied
• Keyboard interactive with a password if it has been supplied or the empty string
• Password if a password has been supplied

I already have this working correctly in my development environment. However, as part of the amazon-ebs builder there is this in the documentation:

Packer will generate a temporary keypair unless ssh_password is used.

This means that although it works in my environment where I am only required to pass an empty keyboard interactive response. In an environment where a password is actually required packer will still skip the generation of the temporary key pair.

I haven't finished the PR yet because it still breaks in that case and I haven't had a chance to look for where that check occurs.

I hope this clarifies my issue.

[rickard-von-essen]

I am not sure I have been clear enough as to my use case here

Thank you for clarifying, I didn't understand that you used publickey _and_ keyboard-interactive. I updated the title to make it more precise.

misunderstanding in packer's SSH code as to how SSH authentication actually works

I wouldn't say that it is a misunderstanding, more a simplification. You are the first one that requests the use of multiple authentication methods.

Nevertheless a possible workaround for your problem could be to use a userscript that configure sshd to use only publickey and as the last step of the provisioning revert that change. (And using temporary ssh key pairs)

[rickard-von-essen]

@mwhooker In some bright future when we change the config format we should consider making a more expressive set of configuration for SSH to handle all variants of authentication.