Dvc: Symlinks on Windows requires elevated permissions

[gamis]

Bug Report

On Windows, with a symlink cache type, dvc checkout and dvc add fail unless you're working from a shell with elevated permissions.

Please provide information about your setup

Output of dvc version:

$ dvc version
DVC version: 1.7.2 (pip)
---------------------------------
Platform: Python 3.7.6 on Windows-10-10.0.18362-SP0
Supports: http, https
Cache types: symlink
Repo: dvc, git

Steps to reproduce

From cmd.exe

mkdir dvctest
cd dvctest
git init
dvc init
dvc config cache.type symlink
echo "test data" > testdata.txt
dvc add testdata.txt -v


2020-09-17 10:47:24,852 DEBUG: Check for update is enabled.
2020-09-17 10:47:24,890 DEBUG: Trying to spawn '['daemon', '-q', 'updater']'
2020-09-17 10:47:25,094 DEBUG: Spawned '['daemon', '-q', 'updater']'
2020-09-17 10:47:25,112 DEBUG: fetched: [(3,)]
2020-09-17 10:47:25,577 DEBUG: Adding 'testdata.txt' to '.gitignore'.
2020-09-17 10:47:25,577 DEBUG: Path 'G:\dvctest1\testdata.txt' inode '1133588046581937827'
2020-09-17 10:47:25,585 DEBUG: fetched: []
2020-09-17 10:47:25,588 DEBUG: Path 'testdata.txt' inode '1133588046581937827'
2020-09-17 10:47:25,588 DEBUG: fetched: []
2020-09-17 10:47:25,588 DEBUG: {'testdata.txt': 'modified'}
2020-09-17 10:47:25,588 DEBUG: Path 'G:\dvctest1\testdata.txt' inode '1133588046581937827'
2020-09-17 10:47:25,588 DEBUG: fetched: [('1600354027790283520', '14', '1388708b11dfee0de78c81fd2a5f9a41', '1600354045588008192')]
2020-09-17 10:47:25,588 DEBUG: Computed stage: 'testdata.txt.dvc' md5: 'None'
2020-09-17 10:47:25,598 DEBUG: Saving 'testdata.txt' to '.dvc\cache\13\88708b11dfee0de78c81fd2a5f9a41'.
2020-09-17 10:47:25,598 DEBUG: Assuming 'G:\dvctest1\.dvc\cache\13\88708b11dfee0de78c81fd2a5f9a41' is unchanged since it is read-only
2020-09-17 10:47:25,598 DEBUG: Cache type 'symlink' is not supported: failed to symlink
2020-09-17 10:47:25,598 DEBUG: Removing 'G:\dvctest1\.dvc\cache\.cache_type_test_file'
Adding...
2020-09-17 10:47:25,608 DEBUG: fetched: [(1,)]
2020-09-17 10:47:25,613 ERROR: no possible cache types left to try out.
------------------------------------------------------------
Traceback (most recent call last):
  File "g:\dvctest\venv\lib\site-packages\dvc\command\add.py", line 22, in run
    external=self.args.external,
  File "g:\dvctest\venv\lib\site-packages\dvc\repo\__init__.py", line 51, in wrapper
    return f(repo, *args, **kwargs)
  File "g:\dvctest\venv\lib\site-packages\dvc\repo\scm_context.py", line 4, in run
    result = method(repo, *args, **kw)
  File "g:\dvctest\venv\lib\site-packages\dvc\repo\add.py", line 96, in add
    stage.commit()
  File "g:\dvctest\venv\lib\site-packages\funcy\decorators.py", line 39, in wrapper
    return deco(call, *dargs, **dkwargs)
  File "g:\dvctest\venv\lib\site-packages\dvc\stage\decorators.py", line 36, in rwlocked
    return call()
  File "g:\dvctest\venv\lib\site-packages\funcy\decorators.py", line 60, in __call__
    return self._func(*self._args, **self._kwargs)
  File "g:\dvctest\venv\lib\site-packages\dvc\stage\__init__.py", line 421, in commit
    out.commit()
  File "g:\dvctest\venv\lib\site-packages\dvc\output\base.py", line 286, in commit
    self.cache.save(self.path_info, self.cache.tree, self.hash_info)
  File "g:\dvctest\venv\lib\site-packages\dvc\cache\base.py", line 330, in save
    self._save(path_info, tree, hash_info, save_link, **kwargs)
  File "g:\dvctest\venv\lib\site-packages\dvc\cache\base.py", line 339, in _save
    self._save_file(path_info, tree, hash_info, save_link, **kwargs)
  File "g:\dvctest\venv\lib\site-packages\dvc\cache\base.py", line 208, in _save_file
    path_info
  File "g:\dvctest\venv\lib\site-packages\dvc\cache\base.py", line 251, in _cache_is_copy
    self.link(test_cache_file, workspace_file)
  File "g:\dvctest\venv\lib\site-packages\dvc\cache\base.py", line 152, in link
    self._link(from_info, to_info, self.cache_types)
  File "g:\dvctest\venv\lib\site-packages\dvc\cache\base.py", line 159, in _link
    self._try_links(from_info, to_info, link_types)
  File "g:\dvctest\venv\lib\site-packages\dvc\remote\slow_link_detection.py", line 34, in wrapper
    return f(remote, *args, **kwargs)
  File "g:\dvctest\venv\lib\site-packages\dvc\cache\base.py", line 187, in _try_links
    raise DvcException("no possible cache types left to try out.")
dvc.exceptions.DvcException: no possible cache types left to try out.
------------------------------------------------------------
2020-09-17 10:47:25,638 DEBUG: Analytics is enabled.
2020-09-17 10:47:25,638 DEBUG: Trying to spawn '['daemon', '-q', 'analytics', 'C:\\Users\\AmisG01\\AppData\\Local\\Temp\\tmp_5zwthdv']'
2020-09-17 10:47:25,840 DEBUG: Spawned '['daemon', '-q', 'analytics', 'C:\\Users\\AmisG01\\AppData\\Local\\Temp\\tmp_5zwthdv']'

Open cmd.exe as Administrator (elevated privledges)

dvc add textdata.txt

100% Add|█████████████████████████████████████████████████████████████████████████████████████|1/1 [00:00,  2.27file/s]

To track the changes with git, run:

        git add .gitignore testdata.txt.dvc

[efiop]

This is expected, our exe installer even adjusts the policy to allow non-admins to create symlinks. For other packages one needs to enable that manualy. That being said, we should at least provide a nicer error message with hints on how to resolve it.

[shcheklein]

@efiop @jorgeorpinel we should probably add this information here - https://dvc.org/doc/user-guide/running-dvc-on-windows ?

[gamis]

Thanks @efiop! The docs make it sound like the pip install and the exe are equivalent. I was using pip install, but I did have developer-mode enabled for windows, which should be enough to support symbolic links.

I just installed the exe, and it still didn't work.

I then did the steps mentioned in that perforce page you referenced, and after a machine restart, dvc works!

Not sure why the changes made by the exe install weren't enough. This is a corporate workstation, so it's possible it has some security constraints the prevented the changes.

So yeah, hopefully this is just a document change for you all!

Thanks again,
g

[jorgeorpinel]

after a machine restart, dvc works!

I think restarting (or just log off/on) after the installer finished would've been enough 🙂

[jorgeorpinel]

we should probably add this information here - https://dvc.org/doc/user-guide/running-dvc-on-windows

Done @shcheklein, see https://github.com/iterative/dvc.org/pull/1787/files#diff-6aa3ffe5a17e4d3c23ac94dfa60d24ba

The docs make it sound like the pip install and the exe are equivalent

I updated the install doc as well to add a note to clarify about this, but is there a specific place where you see that the methods are equivalent, @gamis ? Thanks

[gamis]

If you click on "Get started" at the top of the home page, and then "Install" on the left, you get https://dvc.org/doc/install, which immediately give you pip install directions, and the buries the windows install in a link to an additional page. When I see a pip install, I often don't bother with anything else. On the Windows Install page, it again highlights conda and pip options, and it doesn't mention anything about the symlink issue, apart from a vague, gray-text note about optimization on the bottom, and even that link doesn't say anything about symlink problems on windows.

To be honest, after using dvc a bit, I think it's much better installed with the exe, rather than pip installed.

I would suggest that the main install page and the windows install page more strongly suggest people use the exe installer. And then maybe in the pip and conda sections add some little warning or note.

Another thing to consider: When the cache strategy fails, stderr just says "No other strategies left". Maybe on windows it could give a hint about the solution.

Thanks all for taking a look at this! I'd hate people to give up on dvc because of little things like this.

[jorgeorpinel]

There's no pip install in https://dvc.org/doc/install 🙂 just a mention of pip down here:

After the links to installation pages per platform

But indeed the Windows installation page doesn't feature the info about symlinks either (just links to the doc that has it), and has the exe installer all the way in the bottom.

after using dvc a bit, I think it's much better installed with the exe, rather than pip

👉 So question, guys: should I feature the symlinks note on the top of the Win installation guide? Or just put the exe installer method (which already has a note about symlinks) on top?

maybe in the pip and conda sections add some little warning or note

Not sure about this, may be TMI. Aren't we expecting most users to stick with the default cache.type? (reflink,copy only)

"No other strategies left". Maybe on windows it could give a hint about the solution

➕ That's a great suggestion, in fact it can apply to all systems. "No other strategies left" is misleading: there probably are others available, but they have to be configured.

[jorgeorpinel]

Aren't we expecting most users to stick with the default cache.type? (reflink,copy only)

I could also put the note about Windows (or link to Win tips doc) in the config doc where you would learn about using symlinks as cache.type.

[gamis]

Yeah, if the default is copy and you think that's going to be the typical use case, then maybe the best place to put this info would be in the config doc and in the warning-out-of-strategies message.