Invoke: Responder simply doesn't work

[DominikSerafin]

I've created a task set to backup a postgres database to directory which I set:

invoke.json

{
    "debug": true,
    "run": {
        "echo": true
    }
}

tasks.py

from invoke import task, run
from invoke.watchers import Responder
import sys, os, re

@task
def db_backup(context, backup_dir):

    DB_NAME = os.getenv('DB_NAME')
    DB_USER = os.getenv('DB_USER')
    DB_PASSWORD = os.getenv('DB_PASSWORD')

    username = ' --username='+DB_USER
    host = ' --host=localhost'
    dbname = ' --dbname='+DB_NAME
    output_file = '"'+backup_dir+DB_NAME+'_$(date +\'%y-%m-%d-%H%M\').psql"'

    responder = Responder(pattern=r"Password: ", response=DB_PASSWORD+'\n')
    context.run('pg_dump ' + username + host + dbname + ' > ' + output_file, watchers=[responder])

When I execute it from shell on Ubuntu 16.04 using invoke db_backup /srv/backups/ then it executes without problems except one - the prompt with "Password: " is shown and isn't automatically entered by Responder. I've tried various regexes (like adding more spaces, less spaces, newlines, etc.) but none of them worked.

I'm working on latest Invoke (218feb3b2f53f8b0c0554892a96094eeb2591617)

Am I doing something wrong or is this a bug?

[bitprophet]

Sounds like a bug to me. Can't think offhand what would cause it, it's possible there's a regression - I'm not personally using responder code in my daily so I wouldn't have immediately noticed (though there are a bunch of tests). Could also be some other silly but hard to spot edge case. E.g. unicode vs str (though I'd assume the re module isn't quite that naive...)

Responder is a very small amount of code so you could drop some debug logging in it (it has none by default, at the moment. grump.) and probably figure out what's happening quite fast.

Will add to a milestone so I am reminded to try and repro this next time I cut a release in case you aren't able to identify the problem, pg_dump is one of my friends so I should be able to run exactly what you are. (Only triaging right now so I don't have time to deep dive.)

Thanks for the report!

[DominikSerafin]

Thank you for response. I might try to later identify what's the problem, but for right now I've gotten around it with using PGPASSWORD environment variable (thanks to this) so there isn't any need for manually entering password.

Attaching the code so others with that problem can see solution (although it's hackish and less secure):

from invoke import task, run
import sys, os, re

@task
def db_backup(context, backup_dir):

    DB_NAME = os.getenv('DB_NAME')
    DB_USER = os.getenv('DB_USER')
    DB_PASSWORD = os.getenv('DB_PASSWORD')

    os.environ.setdefault('PGPASSWORD', DB_PASSWORD)

    username = ' --username='+DB_USER
    host = ' --host=localhost'
    dbname = ' --dbname='+DB_NAME
    no_password_prompt = ' --no-password'
    output_file = '"'+backup_dir+DB_NAME+'_$(date +\'%y-%m-%d-%H%M\').psql"'

    context.run('pg_dump ' + no_password_prompt + username + host + dbname + ' > ' + output_file)

[acdimalev]

This is an instance that requires pty=True. Just tested to confirm.

http://www.pyinvoke.org/faq.html#running-local-shell-commands-run

[bitprophet]

@acdimalev Meaning the issue is simply that pg_dump isn't even surfacing the prompt when run w/o a controlling PTY? That'd certainly cause this...

[acdimalev]

More specifically it seems to be this issue.

http://www.pyinvoke.org/faq.html#the-auto-responder-functionality-isn-t-working-for-my-password-prompts

I tested by first redirecting both stdout and stderr to /dev/null from a command line (i.e. 1>/dev/null 2>/dev/null), and still observing a password prompt. Then tested with the script in the initial post first without then with pty=True.

Peeking at the system calls from pg_dump, this makes sense.

open("/dev/tty", O_WRONLY|O_CREAT|O_TRUNC, 0666) = 4
fstat(4, {st_mode=S_IFCHR|0666, st_rdev=makedev(5, 0), ...}) = 0
ioctl(4, TCGETS, {B38400 opost isig icanon -echo ...}) = 0
write(4, "Password: ", 10)              = 10

[bitprophet]

Oh, hah, I'd personally already forgotten all about that FAQ. Ugh. Glad I noticed it at the time I guess. Also glad you figured it out - thanks for commenting! Will close, hopefully @DominikSerafin notices this eventually :)