Pyinfra: Using su_user with use_su_login yields an error with a macOS target

[harold-b]

Describe the bug
Using su_user with use_su_login yields an error with a macOS target. Using use_su_login alone works without specifying the user, but it does not seem to load and shell profiles (nor .zshrc or .bash_profile):

-> Proposed changes:
    Groups: mac_vms
    [vm]   Operations: 1   Commands: 1

--> Beginning operation run...
--> Starting operation: Test ruby ("commands=['which ruby']",)
    [pyinfra\api\operations] Starting operation Test ruby on vm
    [pyinfra\api\connectors\ssh] Running command on vm: (pty=None) su -l the_user -s `which sh` -c 'which ruby'
[vm] >>> su -l the_user -s `which sh` -c 'which ruby'
[vm] su: Sorry
    [pyinfra\api\connectors\ssh] Waiting for exit status...
    [pyinfra\api\connectors\ssh] Command exit status: 1
    [vm] Error
    [pyinfra\api\state] Failing hosts: vm
No hosts remaining!
--> ←[31m←[1mpyinfra error←[0m:

To bring some context:
I have a target macOS machine with chruby set up using a specific version of ruby, under .rubies built with ruby-install. I want to install some gems on the target machine with pyinfra, but this requires it to have chruby run beforehand so that it selects the correct gem version, instead of the one pre-installed with macOS. The commands to do this are in the .zshrc, and when I login to an SSH shell it works as intended. However, even when running with use_su_login = True, it returns the incorrect version of ruby:

server.shell(
    name = 'Test ruby'
    ,use_su_login = True
    ,commands = [ 'which ruby' ]
)

# Results in:
[vm] >>> sh -c 'which ruby'
[vm] /usr/bin/ruby

From an SSH login shell:

> which ruby
/Users/USER/.rubies/ruby-2.7.1/bin/ruby

To Reproduce
With a macOS target (my target is Catalina 10.15.6) such an operation with both su_user and use_su_login specified:

server.shell(
    name = 'Test ruby'
    ,su_user      = host.data.user
    ,use_su_login = True
    ,commands     = [ 'which ruby' ]
)

Expected behavior
No errors to occur

Meta

• Include output of pyinfra --support.
  Note that I am on 1.0.2 as I ran into some other issues with since 1.0.3. I plan to report the issues as well as soon as I resolve this so I can continue with the other ops.

System: Windows
      Platform: Windows-10-10.0.19041-SP0
      Release: 10
      Machine: AMD64
    pyinfra: v1.0.2
    Executable: C:\Users\the_user\AppData\Local\Programs\Python\Python38-32\Scripts\pyinfra.exe
    Python: 3.8.3 (CPython, MSC v.1925 32 bit (Intel))

• How was pyinfra installed (source/pip)?
  pip

Thanks!

[Fizzadar]

@harold-b I think there's a couple of things going on here - in your first example I believe this is an issue with su needing a password which pyinfra cannot provide (unfortunately su, unlike sudo, does not support passing an askpass program). This is poorly highlighted by the [vm] su: Sorry output (needs improvement).

I believe it should be possible to achieve this by using sudo + sudo_user + use_sudo_login; I have tested this locally and it works w/pyinfra @local server.shell 'echo $HOME' sudo=true sudo_user=root use_sudo_login=true -v.

The second example:

server.shell(
    name = 'Test ruby'
    ,use_su_login = True
    ,commands = [ 'which ruby' ]
)

This is expected - without su_user, use_su_login has no effect (would be useful to show a warning about this, noted in https://github.com/Fizzadar/pyinfra/issues/414).

[harold-b]

Thanks very much for the info.

Is there no way then to be able to use the Gem operations in such a way that it would load the user's .*rc files? I did end up using a workaround to install the gems but I was hoping to make use of the Gem operations, but with the customized ruby distribution.

This is expected - without su_user, use_su_login has no effect (would be useful to show a warning about this, noted in #414).

Ah I see. I was expecting it to work since I have the user configured in the SSH config file for that host (running that same operation but with the command whoami does return the correct user).

Thanks again for your help.

[harold-b]

Note that I opened a new issue #415 based on my comment on the original post re: Why using 1.0.2.

[Fizzadar]

Thanks for making that other issue @harold-b!

I believe you can still add sudo here to bypass the su password prompt? ie:

server.shell(
    name='Test ruby',
    sudo=True,
    use_sudo_password=True,  # might not be required depending on sudo config!
    su_user='my_user',
    use_su_login=True,
)

OR - using sudo with a user direct could also work and basically don't use su at all:

server.shell(
    name='Test ruby',
    sudo=True,
    use_sudo_password=True,  # might not be required depending on sudo config!
    sudo_user='my_user',
    use_sudo_login=True,
)

I've tested both variants locally and they work as expected (and load bashrc/etc properly). If these still fail could you test the same but direct via SSH, ie ssh user@host "which ruby" which might point towards the issue.

[harold-b]

Sorry about that, for some reason I forgot that all these *sudo/su * parameters apply to all operations, not just server ones 😮. (Because i wanted to use the gem.packages op.)

Your second version worked well! :) Unfortunately the first version with su_user and use_su_login hangs indefinitely after applying the sudo password :(

But the second one installed the packages with the correct user-configured gem and repository.

[Fizzadar]

Glad to hear it works :) I'll see if I can get a test for the hanging one as that shouldn't happen!

[Fizzadar]

So the reason for the first hanging is su behaves differently on MacOS vs. Linux; ie:

# Mac, `-c` ignored and drops into a shell (or hangs over pyinfra due to no shell)
sudo -H -k su -l root -s `which sh` -c 'echo hi'
Password:
Nicks-iMac:~ root# 

# Linux, `-c` works fine:
vagrant@vagrant:~$ sudo -H -k su -l root -s `which sh` -c 'echo hi'
hi

[harold-b]

Interesting, looks like you found the culprit. Thanks for looking into this as well.

[Fizzadar]

This should now be fixed by https://github.com/Fizzadar/pyinfra/commit/0c7234be0c3d02172da63f843260851b7ff66794! I've released this immediately as 1.1.dev2 and, assuming no further issues, will release 1.1 this week :)

[Fizzadar]

This has now been released in v1.1 🎉!

[harold-b]

Excellent! Thanks for your hard work. I will have a try with the new version as soon as I can and report back for confirmation.

[harold-b]

I tested it and it no longer hangs :)
FYI, however, that this method (the use_su_login version) does not appear to use a logins shell, in the testing it still points to the pre-installed ruby

[Fizzadar]

Thanks for testing @harold-b - I've added https://github.com/Fizzadar/pyinfra/issues/426 to track the login shell issue 😬