Invoke: Invoke 0.13 breaks on Windows (unavailable default shell)

[MinchinWeb]

If I define a simple task, with the newly required context:

@task
def test(ctx):
    ctx.run('dir')

This task should display a directory listing. Instead (on Windows 10, Python 3.5.2, Invoke 0.13), it breaks and displays the following traceback:

Traceback (most recent call last):
  File "c:\program files\python 3.5\lib\runpy.py", line 184, in _run_module_as_main
    "__main__", mod_spec)
  File "c:\program files\python 3.5\lib\runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "C:\Program Files\Python 3.5\Scripts\invoke.exe\__main__.py", line 9, in <module>
  File "c:\program files\python 3.5\lib\site-packages\invoke\program.py", line 270, in run
    self.execute()
  File "c:\program files\python 3.5\lib\site-packages\invoke\program.py", line 381, in execute
    executor.execute(*self.tasks)
  File "c:\program files\python 3.5\lib\site-packages\invoke\executor.py", line 113, in execute
    result = call.task(*args, **call.kwargs)
  File "c:\program files\python 3.5\lib\site-packages\invoke\tasks.py", line 111, in __call__
    result = self.body(*args, **kwargs)
  File "S:\Documents\GitHub\blog.minchin.ca-pelican\tasks.py", line 106, in test
    ctx.run('dir')
  File "c:\program files\python 3.5\lib\site-packages\invoke\context.py", line 53, in run
    return runner_class(context=self).run(command, **kwargs)
  File "c:\program files\python 3.5\lib\site-packages\invoke\runners.py", line 259, in run
    self.start(command, shell, env)
  File "c:\program files\python 3.5\lib\site-packages\invoke\runners.py", line 966, in start
    stdin=PIPE,
  File "c:\program files\python 3.5\lib\subprocess.py", line 947, in __init__
    restore_signals, start_new_session)
  File "c:\program files\python 3.5\lib\subprocess.py", line 1224, in _execute_child
    startupinfo)
FileNotFoundError: [WinError 3] The system cannot find the path specified

I believe this is because the default shell (/bin/bash) in unavailable, at least by that name, on the default Windows CMD. If you try to run /bin/bash on Windows CMD, it will display a similar error message ("The system cannot find the path specified").

Setting the shell on the run command with the full path of CMD seems to fix it. I.e. the following works as expected:

@task
def test(ctx):
    ctx.run('dir', shell="C:\Windows\system32\cmd.EXE")

Setting the shell to just the name of the exacutable ("cmd" or "cmd.exe") don't work.

• So, how to I change the default shell in my default context? (I couldn't find that searching the docs)
• Can Invoke have a Windows native default shell when running on Windows? (CMD or Powershell, I'm not particularly picky; I just want it to work out of the box)
• If the default shell is going to remain unchanged, can the documentation be updated to note that the defaults break on Windows, and how to work around it?

Thanks!

[bitprophet]

See #345 which I think is the same problem - one of the comments notes how you can set the shell override globally, too :)

[mkusz]

The best way to fix it on windows is to put inside tasks.py:

from sys import platform
from os import environ
from invoke import Collection, task

@task
def testing(ctx):
    ctx.run("echo")

ns=Collection()
ns.add_task(testing)

# Workaround for Windows execution
if platform == 'win32':
    ns.configure({'run': {'shell': environ['COMSPEC']}})`  # This is path to cmd.exe

Then after executing invoke testing you should se ECHO is on. which means it's working ok.
PS. In fact this should be put somewhere inside documentation.

[NickVolynkin]

@mkusz is there a way to use this workaround without explicitly adding all tasks to a Collection?

[genericmoniker]

@NickVolynkin I find it easier (no changes required to tasks.py) to create a .invoke.yaml file in your user directory and set the run config there:

    run:
        shell: C:\Windows\System32\cmd.exe

[NickVolynkin]

@genericmoniker won't it break running tasks on *nix systems?

[mkusz]

@NickVolynkin setting shell using invoke.yaml is not cross system compatible.
If you need to set it without Collection you can do it like this:

from sys import platform
from os import environ
from invoke import task

def shell(ctx):
    if platform == 'win32':
        ctx._config['run']['shell'] = environ['COMSPEC']
    return ctx

@task
def testing(ctx):
    ctx = shell(ctx)  # Call function to setup proper shell for Windows
    ctx.run("echo")

A disadvantage of this is that you need to call shell function inside each task. I was trying to write a decorator but without success (there is something happening in between task decorator and task definition with ctx injecting I wasn't able to solve in reasonable time). You can try to rewrite this solution to use decorator but I don't have time for it.

[genericmoniker]

@NickVolynkin @mkusz

The strategy of using .invoke.yaml is a per-Windows-system fix. It doesn't need to be cross-system compatible, and you just wouldn't do it for *nix systems because it isn't necessary.

Advantage: You don't have to do anything special in tasks.py itself.
Disadvantage: Extra step (creating .invoke.yaml) required for Windows users before they can start using Invoke. If that's not feasible in a given situation, then another solution would be appropriate.

[mkusz]

@genericmoniker Agree that yaml file is easier but it also has some disadvantages like:

• path to cmd.exe is hardcoded (my approach read system variable)
• (like I said before) it's not cross system compliant
  I'm using invoke to execute web tests over Linux and Windows so I need cross-system compatibility.

[thebjorn]

@genericmoniker Sorry if this is a little rant-y, but that extra step of creating an .invoke.yaml file would need to be done, manually, for every Windows user. pip can't write to random locations when installing a new package that uses invoke, meaning that any automation goes right out the window -- and that's sort of the raison d'être for invoke..

Hard coding shell="/bin/bash" is so obviously wrong that I don't know how it got into a package that claims to be x-platform. Would you be ok if shell was set to undefined and had to be specified in .invoke.yaml for every *nix user on every *nix system? (or require sudo privileges even if installed in a virtualenv?) It would seem pretty weird that a package designed around running external programs couldn't find the shell executable -- it's even weirder on Windows, where there is exactly one (1!) way of finding the shell (the COMSPEC environment variable).

Regardless, (1) none of the work-arounds are documented where new Windows users would see them (www.pyinvoke.org or README file), and (2) following the introductory example gives a cryptic error (File not found, without mentioning which file). I therefore still maintain that invoke is broken on windows, especially for new users (but I've been bitten twice myself and I don't consider myself a new invoke user). That it has been broken for 16 months, and the fix requires 5 lines of code, is sending a pretty clear signal to Windows users, and users that need cross platform tools.

[genericmoniker]

@thebjorn No worries -- I'd also love to see the real fix rather than a workaround. I was just trying to share a tip that has worked reasonably well for me in the meantime.