Salt: [salt-ssh, 3001.2, 3001.3, 3002.1] ssh password authentication seems to be broken

[victorenr-bit]

Description
I can no longer use salt-ssh to execute states after upgrading to salt-ssh 3001.3 or 3002.1, previous versions do work (3001.2 and 3002). Trying to execute test.version on one of the hosts in the roster file I get:

$ salt-ssh -i 'sta-dca-payments-us' test.version -l debug

salt@6d7841e67085:~$ salt-ssh -i 'sta-dca-payments-us' test.version -l debug
[INFO    ] Loading Saltfile from '/home/salt/Saltfile'
[DEBUG   ] Reading configuration from /home/salt/Saltfile
[DEBUG   ] Reading configuration from /home/salt/config/master
[DEBUG   ] Configuration file path: /home/salt/config/master
[WARNING ] Insecure logging configuration detected! Sensitive data may be logged.
[DEBUG   ] LazyLoaded flat.targets
[DEBUG   ] LazyLoaded jinja.render
[DEBUG   ] LazyLoaded yaml.render
[DEBUG   ] compile template: ./config/roster
[DEBUG   ] LazyLoaded gpg.render
[DEBUG   ] Results of YAML rendering: 
OrderedDict([('sta-dca-payments-us', OrderedDict([('host', 'sta-dca-payments-us.company.com'), ('passwd', '-----BEGIN PGP MESSAGE-----EDITED -----END PGP MESSAGE-----\n')])), ('sta-dca-payments-db', OrderedDict([('host', 'sta-dca-payments-db.company.com'), ('passwd', '-----BEGIN PGP MESSAGE-----EDITED-----END PGP MESSAGE-----\n')])), ('sta-dca-payments-db-replica', OrderedDict([('host', 'sta-dca-payments-db-replica.company.com'), ('passwd', '-----BEGIN PGP MESSAGE-----EDITED-----END PGP MESSAGE-----\n')])), ('sta-dca-payments-logs', OrderedDict([('host', 'sta-dca-payments-logs.company.com'), ('passwd', '-----BEGIN PGP MESSAGE-----EDITED-----END PGP MESSAGE-----\n')]))])
[PROFILE ] Time (in seconds) to render './config/roster' using 'yaml' renderer: 0.005864858627319336
[DEBUG   ] Reading GPG keys from: /secret-storage
[PROFILE ] Time (in seconds) to render './config/roster' using 'gpg' renderer: 0.12304568290710449
[DEBUG   ] LazyLoaded roster_matcher.targets
[DEBUG   ] Matched minions: {'sta-dca-payments-us': {'user': 'deployment-user', 'sudo': True, 'tty': True, 'host': 'sta-dca-payments-us.company.com', 'passwd': "EDITING THIS PASSWORD BUT IT APPEARS TO BE CORRECT"}}
Saving key "/home/salt/pki_dir/ssh/salt-ssh.rsa" failed: passphrase is too short (minimum five characters)
[DEBUG   ] LazyLoaded roots.envs
[DEBUG   ] Could not LazyLoad roots.init: 'roots.init' is not available.
[DEBUG   ] Updating roots fileserver cache
[DEBUG   ] LazyLoaded local_cache.prep_jid
[DEBUG   ] Adding minions for job 20201111134559415761: ['sta-dca-payments-us']
[DEBUG   ] Could not LazyLoad test.version: 'test.version' is not available.
[DEBUG   ] Performing shimmed, blocking command as follows:
test.version
[DEBUG   ] Executing command: ssh sta-dca-payments-us.company.com -t -t -o KbdInteractiveAuthentication=no -o PasswordAuthentication=yes -o GSSAPIAuthentication=no -o ConnectTimeout=65 -o StrictHostKeyChecking=no -o Port=22 -o IdentityFile=/home/salt/pki_dir/ssh/salt-ssh.rsa -o User=deployment-user  mkdir -p 
[DEBUG   ] Child Forked! PID: 130  STDOUT_FD: 12  STDERR_FD: 14
[DEBUG   ] Terminal Command: ssh sta-dca-payments-us.company.com -t -t -o KbdInteractiveAuthentication=no -o PasswordAuthentication=yes -o GSSAPIAuthentication=no -o ConnectTimeout=65 -o StrictHostKeyChecking=no -o Port=22 -o IdentityFile=/home/salt/pki_dir/ssh/salt-ssh.rsa -o User=deployment-user mkdir -p
[DEBUG   ] Executing command: scp -o KbdInteractiveAuthentication=no -o PasswordAuthentication=yes -o GSSAPIAuthentication=no -o ConnectTimeout=65 -o StrictHostKeyChecking=no -o Port=22 -o IdentityFile=/home/salt/pki_dir/ssh/salt-ssh.rsa -o User=deployment-user  /tmp/shim_07zynizf sta-dca-payments-us.company.com:.5849daacfc5a.py
[DEBUG   ] Child Forked! PID: 131  STDOUT_FD: 12  STDERR_FD: 14
[DEBUG   ] Terminal Command: scp -o KbdInteractiveAuthentication=no -o PasswordAuthentication=yes -o GSSAPIAuthentication=no -o ConnectTimeout=65 -o StrictHostKeyChecking=no -o Port=22 -o IdentityFile=/home/salt/pki_dir/ssh/salt-ssh.rsa -o User=deployment-user /tmp/shim_07zynizf sta-dca-payments-us.company.com:.5849daacfc5a.py
[DEBUG   ] Executing command: ssh sta-dca-payments-us.company.com -t -t -o KbdInteractiveAuthentication=no -o PasswordAuthentication=yes -o GSSAPIAuthentication=no -o ConnectTimeout=65 -o StrictHostKeyChecking=no -o Port=22 -o IdentityFile=/home/salt/pki_dir/ssh/salt-ssh.rsa -o User=deployment-user  /bin/sh '$HOME/.5849daacfc5a.py'
[DEBUG   ] Child Forked! PID: 133  STDOUT_FD: 12  STDERR_FD: 14
[DEBUG   ] Terminal Command: ssh sta-dca-payments-us.company.com -t -t -o KbdInteractiveAuthentication=no -o PasswordAuthentication=yes -o GSSAPIAuthentication=no -o ConnectTimeout=65 -o StrictHostKeyChecking=no -o Port=22 -o IdentityFile=/home/salt/pki_dir/ssh/salt-ssh.rsa -o User=deployment-user /bin/sh  '$HOME/.5849daacfc5a.py'
[DEBUG   ] Executing command: ssh sta-dca-payments-us.company.com -t -t -o KbdInteractiveAuthentication=no -o PasswordAuthentication=yes -o GSSAPIAuthentication=no -o ConnectTimeout=65 -o StrictHostKeyChecking=no -o Port=22 -o IdentityFile=/home/salt/pki_dir/ssh/salt-ssh.rsa -o User=deployment-user  rm '$HOME/.5849daacfc5a.py'
[DEBUG   ] Child Forked! PID: 134  STDOUT_FD: 12  STDERR_FD: 14
[DEBUG   ] Terminal Command: ssh sta-dca-payments-us.company.com -t -t -o KbdInteractiveAuthentication=no -o PasswordAuthentication=yes -o GSSAPIAuthentication=no -o ConnectTimeout=65 -o StrictHostKeyChecking=no -o Port=22 -o IdentityFile=/home/salt/pki_dir/ssh/salt-ssh.rsa -o User=deployment-user rm $HOME/.5849daacfc5a.py
[DEBUG   ] RETCODE sta-dca-payments-us.company.com: 127
[DEBUG   ] SHIM retcode(127) and command: Password:
[DEBUG   ] LazyLoaded nested.output
sta-dca-payments-us:
    ----------
    retcode:
        127
    stderr:
        no such identity: /home/salt/pki_dir/ssh/salt-ssh.rsa: No such file or directory
        Connection to sta-dca-payments-us.company.com closed.
    stdout:
        Password: 
        /bin/sh: 0: Can't open $HOME/.5849daacfc5a.py

Setup
/home/salt/Saltfile

salt-ssh:
  roster_file: ./config/roster
  config_dir: ./config
  ssh_max_procs: 30
  ssh_wipe: True
  ssh_log_file: /home/salt/salt-ssh.log
  log_file: /home/salt/salt.log

/home/salt/config/master:

cachedir: /home/salt/cachedir
yaml_utf8: True
pki_dir: /home/salt/pki_dir
pillar_opts: True
roster_defaults:
  user: deployment-user
  sudo: True
  tty: True
file_roots:
  base:
    - /home/salt/saltstack/salt
pillar_roots:
  sta:
    - /home/salt/saltstack/pillar/sta
gpg_keydir: /secret-storage

/home/salt/config/roster:

#!yaml|gpg
# -*- coding: utf-8 -*-
# vim: ft=yaml
---
sta-dca-payments-us:
  host: sta-dca-payments-us.company.com
  passwd: |
    -----BEGIN PGP MESSAGE-----

    -----END PGP MESSAGE-----

(This file includes more hosts with this same structure)

Steps to Reproduce the behavior
See the initial description, it happens with any state/module.

Expected behavior
I expect salt-ssh to execute state/modules like it used to.

Versions Report

salt --versions-report

(Provided by running salt --versions-report. Please also mention any differences in master/minion versions.)

Salt Version:
           Salt: 3001.3

Dependency Versions:
           cffi: Not Installed
       cherrypy: Not Installed
       dateutil: Not Installed
      docker-py: Not Installed
          gitdb: Not Installed
      gitpython: Not Installed
         Jinja2: 2.11.2
        libgit2: Not Installed
       M2Crypto: Not Installed
           Mako: Not Installed
   msgpack-pure: Not Installed
 msgpack-python: 1.0.0
   mysql-python: Not Installed
      pycparser: Not Installed
       pycrypto: Not Installed
   pycryptodome: Not Installed
         pygit2: Not Installed
         Python: 3.6.9 (default, Oct  8 2020, 12:12:24)
   python-gnupg: Not Installed
         PyYAML: 5.3.1
          PyZMQ: Not Installed
          smmap: Not Installed
        timelib: Not Installed
        Tornado: 4.5.3
            ZMQ: Not Installed

System Versions:
           dist: ubuntu 18.04 bionic
         locale: ANSI_X3.4-1968
        machine: x86_64
        release: 5.6.15-050615-lowlatency
         system: Linux
        version: Ubuntu 18.04 bionic

Additional context
salt-ssh is executed inside a docker container, our image is based on ubuntu bionic with salt-ssh installed with pip basically is something like the following:

FROM ubuntu:bionic

ENV DEBIAN_FRONTEND=noninteractive
ENV BUILD_DEPENDENCIES python3-pip \
    dpkg-dev \
        g++ \
        gcc \
        libc6-dev \
        make \
        python3-dev \
        python3-setuptools \
        python3-wheel

ENV GPG_PACKAGES gnupg gpg-agent

ENV USEFUL_TOOLS vim less

RUN apt-get update && \
apt-get install --no-install-recommends -y \
        $GPG_PACKAGES \
    $USEFUL_TOOLS \
        locales \
        openssh-client \
    sshpass \
    $BUILD_DEPENDENCIES && \
LANG="C.UTF-8" pip3 install salt-ssh==3001.3 && \
apt-get remove --purge -y $BUILD_DEPENDENCIES && \
apt-get clean && rm -rf /var/lib/apt/lists/*

ENV LANG en_US.UTF-8
ENV LANGUAGE ""
ENV LC_CTYPE "en_US.UTF-8"
ENV LC_NUMERIC en_US.UTF-8
ENV LC_TIME en_US.UTF-8
ENV LC_COLLATE "en_US.UTF-8"
ENV LC_MONETARY en_US.UTF-8
ENV LC_MESSAGES "en_US.UTF-8"
ENV LC_PAPER en_US.UTF-8
ENV LC_NAME en_US.UTF-8
ENV LC_ADDRESS en_US.UTF-8
ENV LC_TELEPHONE en_US.UTF-8
ENV LC_MEASUREMENT en_US.UTF-8
ENV LC_IDENTIFICATION en_US.UTF-8
ENV LC_ALL ""
RUN useradd -m salt -s /bin/bash
COPY Saltfile /home/salt/Saltfile
RUN chown salt:salt /home/salt/Saltfile
USER salt:salt
RUN mkdir -p /home/salt/config \
             /home/salt/cachedir \
             /home/salt/pki_dir \
             /home/salt/.bin \
             /home/salt/saltstack \
             /home/salt/saltstack/salt \
             /home/salt/saltstack/pillar
COPY launch-salt.sh /home/salt/.bin/launch-salt.sh
COPY master /home/salt/config/master
COPY roots_config /roots_config
WORKDIR /home/salt

ENTRYPOINT ["/home/salt/.bin/launch-salt.sh"]

Our launch-salt.sh setups gpg based on some based bind mounts and setups the environment for salt-ssh to work, but this has nothing to do with the issue, since installing a previous version fixes it. I am willing to provide more details if needed.

[victorenr-bit]

It looks like salt-ssh is not passing the password properly to the ssh commands, since it seems to be displaying the password prompt, previous versions also don't complain about the lack of the .rsa file.

[Ch3LL]

no such identity: /home/salt/pki_dir/ssh/salt-ssh.rsa: No such file or directory

What is in that directory? Even if i delete that file salt-ssh recreates that rsa key. Are the perms correct?

Also, I have not been able to replicate this using your config files yet. What version was it working on previously?

[victorenr-bit]

Strangely it seems that with previous versions that keypair was populated automatically. But with the new version it is not created. I get:

salt@5c7a7f65defb:~$ salt-ssh -i 'sta-*-logs' state.apply test=True
Saving key "/home/salt/pki_dir/ssh/salt-ssh.rsa" failed: passphrase is too short (minimum five characters)
sta-dca-payments-logs:
    ----------
    _error:
        Failed to return clean data
    retcode:
        127
    stderr:
        no such identity: /home/salt/pki_dir/ssh/salt-ssh.rsa: No such file or directory
        Connection to sta-dca-payments-logs.company.com closed.
    stdout:
        Password: 
        /bin/sh: 0: Can't open $HOME/.e12f4fd3e406.py

and yes I can write inside that folder, for example:

salt@5c7a7f65defb:~/pki_dir$ ls
ssh
salt@5c7a7f65defb:~/pki_dir$ cd ssh/
salt@5c7a7f65defb:~/pki_dir/ssh$ ls
salt@5c7a7f65defb:~/pki_dir/ssh$ touch 1
salt@5c7a7f65defb:~/pki_dir/ssh$ ls
1
salt@5c7a7f65defb:~/pki_dir/ssh$ ls -l
total 0
-rw-r--r-- 1 salt salt 0 Nov 12 16:18 1

By the way that folder is setup in the config file (config/master) it's the pki_dir variable, if I remove it I get:

salt@5c7a7f65defb:~$ salt-ssh -i 'sta-*-logs' state.apply test=True
salt-ssh could not be run because it could not generate keys.

You can probably resolve this by executing this script with increased permissions via sudo or by running as root.
You could also use the '-c' option to supply a configuration directory that you have permissions to read and write to.

So it seems like it is required by salt-ssh.

Anyway if I use ssh-keygen -t rsa to create a keypair in that directory:

salt@5c7a7f65defb:~$ ssh-keygen -t rsa
Generating public/private rsa key pair.
Enter file in which to save the key (/home/salt/.ssh/id_rsa): /home/salt/pki_dir/ssh/salt-ssh.rsa
Enter passphrase (empty for no passphrase): 
Enter same passphrase again: 
Your identification has been saved in /home/salt/pki_dir/ssh/salt-ssh.rsa.
Your public key has been saved in /home/salt/pki_dir/ssh/salt-ssh.rsa.pub.
The key fingerprint is:
SHA256:usHrBT0KcQ6ppWCTOGo9gTGrUpeRrWEZfKFd7sjr9mc salt@5c7a7f65defb
The key's randomart image is:
+---[RSA 2048]----+
| o .o=...        |
|. * **oo         |
|+*.oO+o .        |
|++o=o* +         |
|+.oo. = S        |
|o   .o = .       |
|      * .        |
|     ..=  E      |
|     o=..o       |
+----[SHA256]-----+

and then I execute salt-ssh again:

salt@5c7a7f65defb:~$ salt-ssh -i 'sta-*-logs' state.apply test=True
sta-dca-payments-logs:
    ----------
    _error:
        Failed to return clean data
    retcode:
        127
    stderr:
        Connection to sta-dca-payments-logs.company.com closed.
    stdout:
        Password: 
        /bin/sh: 0: Can't open $HOME/.a0a26193e902.py

I still see the "Password:" stuff on stdout, previous versions that worked without this issue are 3001.2 and 3002 for example.

[Ch3LL]

okay looks like i was able to replicate it now, thanks for the additional information, i will look more into this.

[victorenr-bit]

Thanks, just let me know if you require any other test/information.

[victorenr-bit]

I did another test just to clarify in which versions the issue is happening, 3001.2 also fails, 3002 and 3001.1 both work (I built a fresh image to discard a failure in any other component).

[Ch3LL]

can you give the fix in https://github.com/saltstack/salt/pull/58948 a try

[victorenr-bit]

can you give the fix in #58948 a try

That change fixed the issue (Thanks!), I am still getting this message though (I am not sure if it is just cosmetic or if I should care about it at all, also I am observing a change in behavior, since previous versions used to create that salt-ssh ssh keypair, but new ones don't):

salt@b316bcde7603:~$ salt-ssh -i 'sta-*-logs' test.version
Saving key "/home/salt/pki_dir/ssh/salt-ssh.rsa" failed: passphrase is too short (minimum five characters)
sta-dca-payments-logs:
    3002.1

[Ch3LL]

i'm not able to replicate that particular error.

What happens if you run the entire command manually ssh-keygen -P "" -f /home/salt/pki_dir/ssh/salt-ssh.rsa -t rsa -q

also what version of openssh are you using?

[victorenr-bit]

ssh-keygen -P "" -f /home/salt/pki_dir/ssh/salt-ssh.rsa -t rsa -q

With the container just created (before running any salt-ssh command at all) I get:

salt@5bdeef8ef60b:~$ ssh-keygen -P "" -f /home/salt/pki_dir/ssh/salt-ssh.rsa -t rsa -q
Saving key "/home/salt/pki_dir/ssh/salt-ssh.rsa" failed: No such file or directory

[Ch3LL]

and what version of openssh are you running?

[victorenr-bit]

and what version of openssh are you running?

oh sorry I forgot about the version (multitasking sucks!):

salt@5bdeef8ef60b:~$ dpkg -l | grep ssh
ii  openssh-client             1:7.6p1-4ubuntu0.3                  amd64        secure shell (SSH) client, for secure access to remote machines
ii  sshpass                    1.06-1                              amd64        Non-interactive ssh password authentication

also

salt@5bdeef8ef60b:~$ ssh -V
OpenSSH_7.6p1 Ubuntu-4ubuntu0.3, OpenSSL 1.0.2n  7 Dec 2017

[Ch3LL]

oh i just re-read your error. If you are going to run that command without running salt-ssh you need to create the directory first:

mkdir ~/pki_dir/ssh/ -p

Can you re-run after creating the directory? If you still get the same error passphrase is too short its most likely an issue with the version of openssh, since I'm on a newer version 8.4p1-2 and i do not see the error. I will see about building and installing that version as well to see if i can then replicate

[victorenr-bit]

Ok, so just to clarify:

1.- When I start the container the pki_dir folder is empty.
2.- When I run any salt-ssh command it complains with the error I pasted before.
3.- When I create the directory ssh manually inside the pki_dir folder (before launching salt-ssh) I get the following:

salt@d0dde3000188:~$ mkdir pki_dir/ssh -p
salt@d0dde3000188:~$ salt-ssh -i 'sta-*-logs' test.version
Saving key "/home/salt/pki_dir/ssh/salt-ssh.rsa" failed: passphrase is too short (minimum five characters)
sta-dca-payments-logs:
    3002.1

So it is still complaining about it, and there is no keypair inside the pki_dir/ssh folder, 3001.1 used to create everything on it's own.

4.- If I manually create the folder and the keypair, then the error is gone in salt-ssh, I can do it on the Dockerfile, but which software is supposed to be responsible of doing it? Does salt-ssh resort to creating it's own keypair if there is none?.

[victorenr-bit]

oh i just re-read your error. If you are going to run that command without running salt-ssh you need to create the directory first:

mkdir ~/pki_dir/ssh/ -p

Can you re-run after creating the directory? If you still get the same error passphrase is too short its most likely an issue with the version of openssh, since I'm on a newer version 8.4p1-2 and i do not see the error. I will see about building and installing that version as well to see if i can then replicate

If I use ubuntu focal instead of bionic in the Dockerfile, the issue seems to be gone here as well, that also works for me. By the way both the pki_dir/ssh folder and the keypair are also auto-created when using focal instead of bionic, weird stuff.

Thanks for your help again.

[Ch3LL]

glad you got it figured out. thanks for your patience with all my questions as well :) Looks like focal does have a newer version of openssh 8.2, so this error is an upstream issue that has been resolved. I will go ahead and close this issue now but let me know if i need to re-open for any reason.

[Ch3LL]

just as a follow up here. the fix here: https://github.com/saltstack/salt/pull/58871/files#diff-2b0018ba85b85d07125591f26aa82ce3f90bbaff1fe966aa83b3ac1113396a3eR33 should resolve the key error as well.

[victorenr-bit]

just as a follow up here. the fix here: https://github.com/saltstack/salt/pull/58871/files#diff-2b0018ba85b85d07125591f26aa82ce3f90bbaff1fe966aa83b3ac1113396a3eR33 should resolve the key error as well.

Nice, so that should fix the issue when using bionic instead of focal, right? I will give it a try later.

[Ch3LL]

yep it should :) let me know

[victorenr-bit]

yep it should :) let me know

Yep it works here :+1: