dvc: possible path translation problems on mingw

[efiop]

$ dvc run -v -d data/Posts.xml.tgz -o data/Posts.xml tar zxf data/Posts.xml.tgz -C data/
Debug: updater is not old enough to check for updates
Debug: PRAGMA user_version;
Debug: fetched: [(2,)]
Debug: CREATE TABLE IF NOT EXISTS state (inode INTEGER PRIMARY KEY, mtime TEXT NOT NULL, md5 TEXT NOT NULL, timestamp TEXT NOT NULL)
Debug: CREATE TABLE IF NOT EXISTS state_info (count INTEGER)
Debug: CREATE TABLE IF NOT EXISTS link_state (path TEXT PRIMARY KEY, inode INTEGER NOT NULL, mtime TEXT NOT NULL)
Debug: INSERT OR IGNORE INTO state_info (count) SELECT 0 WHERE NOT EXISTS (SELECT * FROM state_info)
Debug: PRAGMA user_version = 2;
Running command:
        tar zxf data/Posts.xml.tgz -C data/
/c: /c: Is a directory
Debug: SELECT count from state_info WHERE rowid=1
Debug: fetched: [(1,)]
Debug: UPDATE state_info SET count = 1 WHERE rowid = 1
Error: Traceback (most recent call last):
  File "dvc\command\run.py", line 18, in run
  File "dvc\project.py", line 265, in run
  File "dvc\stage.py", line 435, in run
StageCmdFailedError: Stage 'Posts.xml.dvc' cmd tar zxf data/Posts.xml.tgz -C data/ failed

Error: Failed to run command: Stage 'Posts.xml.dvc' cmd tar zxf data/Posts.xml.tgz -C data/ failed

[mjmikulski]

Here is the link to refined question that I posted on stackoverflow.

I looked at stage.py and found that it uses abspath wheras in other places relpath is used. Maybe that's a hint?

This error seems to be connected with resolving absolute path in mingw.

[efiop]

Hi @mjmikulski !

Could you also show us the output of python -V and which python please?

[mjmikulski]

@efiop

I know it sounds strange: there is no python in this console.

My setup:

1. MINGW comes from Git Extensions
2. I use pythons from anaconda environments (I have identical envs on windows workstation and linux servers, so I can prototype on windows and run heavy computations on linux)
3. I installed DVC simply with windows installer.
4. Than I discovered that when I type dvc in mingw it works.

$ dvc -V
0.19.12

Maybe I should install DVC in mingw directly? But there is no pip in mingw.

[efiop]

@mjmikulski Sorry for a late response.

Ah, so you've installed dvc from our binary package for windows, right? That is an important detail. Could you uninstall the binary and install dvc using pip?(pip install dvc) I suspect that since our binary package was specifically built for plain windows, mingw translation might've messed it up.

Btw, I've tried running dvc on anaconda's python on mingw and it worked fine for me.

[mjmikulski]

@efiop This is on my work computer so I will try it on Monday and let you know.

[mjmikulski]

@efiop Sorry for the delay.

I uninstalled dvc from windows. Then I added python from a conda environemnt to my path and installed dvc in mingw using pip install dvc. Now I have:

$ python -V
Python 3.6.6
$ which python
/c/Users/xxx/AppData/Local/conda/conda/envs/my_env/python
$ dvc -V
0.19.15
$ which dvc
/c/Users/xxx/AppData/Local/conda/conda/envs/my_env/Scripts/dvc

but problem still persists.

If you cannot reproduce the issue it may mean that there may be a problem with my mingw installation.

[efiop]

Hm, interesting. And does dvc run -o test 'echo test > test' work for you? Please notice that echo test > test is wrapped in quotes.

[mjmikulski]

Hmm, it prints:

XXX@YYY MINGW32 ~/repos/playground (deploy-dvc)
$ dvc run -o test 'echo test > test'
Running command:
        echo test > test
/c: /c: Is a directory
Error: Failed to run command: Stage 'test.dvc' cmd echo test > test failed

Having any troubles? Hit us up at dvc.org/support, we are always happy to help!

[efiop]

And how about:

$ printenv &> env1.log
$ dvc run -o env2.log 'printenv &> env2.log'

could you please link both env1.log and env2.log here?

[mjmikulski]

Well, I would have to look thoroughly through log and mask some data which is probably considered sensitive by my employer...

Anyway, any command in dvc run fails.

I will try to reproduce this on another windows machine and let you know in a few days.

[karolnawrocki]

Hi @efiop ,

• I'm also using conda environment
• I tried uninstalling Windows binary package and installed DVC via pip install dvc
• I used Cygwin instead of MinGW and it prints:

xxx@yyy/cygdrive/c/Users/xxx/classify
$ dvc run -d data/Posts.xml.tgz -o data/Posts.xml tar zxf data/Posts.xml.tgz -C data/
Running command:
        tar zxf data/Posts.xml.tgz -C data/
Error: Unexpected error: [WinError 3] The system cannot find the path specified

Having any troubles? Hit us up at dvc.org/support, we are always happy to help!

xxx@yyy/cygdrive/c/Users/xxx/classify
$ dvc run -o test 'echo test > test'
Running command:
        echo test > test
Error: Unexpected error: [WinError 3] The system cannot find the path specified

Having any troubles? Hit us up at dvc.org/support, we are always happy to help!

[efiop]

Hi @karolnawrocki !

I can indeed reproduce your issue. Looks like it is a similar problem, where path translations get messed up. I am looking into it right now, it should help with mingw bug as well.

Thank you for the feedback!

-Ruslan

[efiop]

Ok, got it. Looks like cygwin provides $SHELL env variable, which is /usr/bin/bash and is in posix style. Then, when dvc executes the command, it has both posix and nt style paths, which then get translated into nt paths, since conda's python was not built for cygwin. And obviously /usr/bin/bash is no longer valid when converted to nt path. Preparing a patch right now.

[efiop]

@mjmikulski @karolnawrocki Guys, could you please try out upstream version to see if it works for you? I.e.

pip uninstall dvc
pip install git+https://github.com/iterative/dvc.git

[karolnawrocki]

@efiop
Works with Cygwin, thank you!

[efiop]

I've released 0.20.0 with this fix. Please feel free to upgrade.

@mjmikulski I'm closing this issue for now. I understand that you will only be able to check if it works for you on Monday, so please feel free to reopen.

Thank you guys for all the feedback!

[mjmikulski]

I confirm that now it rocks on mingw! Thanks a million for your quick fix.

[efiop]

@mjmikulski Glad it works for you! Thank you so much for the feedback! 🙂