Git: Create symlinks when cloning repos with symlinks

[dandv]

I understand the "symlinks on Windows" problem is non-trivial, and there's been ongoing work to support them. Right now cloning a repo with a file symlink will create a plain text file corresponding to the symlink.

Tested with the portable 64-bit git version 2.3.6.windows.2 in an Admin git-bash prompt on Windows 7.

[elyscape]

This is expected behavior when core.symlinks is set to false. Windows distributions of Git come with that set globally because symlink creation on Windows requires the user to have administrative privileges and the command to be done from an elevated shell (or for UAC to be disabled). If your user account has administrative privileges and you either always use Git from an elevated shell or have UAC disabled, you can enable true symlink support in Git by setting core.symlinks to true in your global .gitconfig.

[dscho]

@dandv I am assuming that @elyscape's comment helped you because it is spot on.

[dandv]

Unless I'm missing a step, symlinks just aren't created when cloning the test repo:

[elyscape]

You made sure to run git config --global core.symlinks true?

[dandv]

@elyscape: Yes, forgot to include that in the repro steps. Updated the screenshot.

[elyscape]

And you're running Git Bash elevated?

[dandv]

I assume so, since I can create files in C:\, which otherwise would be met with a "Permission denied":

[elyscape]

Try going into the repository and running these commands:

git config core.symlinks true
git rm --cached -r .
git reset --hard

[dandv]

The reset command errors like this:

error: unable to create symlink subdir/test.txt (Function not implemented)
error: unable to create symlink test2 (Function not implemented)
fatal: Could not reset index file to revision 'HEAD'.

[elyscape]

I guess that answers our question: Git for Windows does not presently support symlinks at all.

@dscho: Could you reopen this issue? Or should someone make another for the symlink implementation?

[elyscape]

Looks like this is the source of the failure.

[dscho]

Unfortunately, we had to drop the NTFS reparse point-based symlink emulation support when updating to 2.1.0: be148fa8c099009fce062cc1c344aa880ca4c7fa (see https://github.com/msysgit/git/pull/231 for details).

[dandv]

Surely the issue should stay open then?

[dscho]

@dandv please note that while I reopened this ticket as per your request, it is in dear need of developers getting behind it, though.

[dandv]

Thanks. Perhaps getting attention from StackOverflow users would help. Some have posted some pretty in-depth workarounds and may be interesting in helping.

• http://stackoverflow.com/questions/5917249/git-symlinks-in-windows/
• http://stackoverflow.com/questions/19845607/how-do-i-commit-a-windows-symlink-into-git
• http://stackoverflow.com/questions/11662868/what-happens-when-i-clone-a-repository-with-symlinks-on-windows

[dscho]

Sorry, I fear that StackOverflow users are not necessarily in the best position to take that topic branch introducing symlink support and making it more robust... We really need real developers with a real incentive to work on this.

[elieux]

symlink creation on Windows requires the user to have administrative privileges and the command to be done from an elevated shell (or for UAC to be disabled)

Just to give a complete picture, this is configurable by the way of assigning SeCreateSymbolicLinkPrivilege.

[dscho]

@elieux good point. Maybe it would be a good idea to create a page on our wiki: https://github.com/git-for-windows/git/wiki (maybe "symlinks"?). That way, @dandv could add information vetted from StackOverflow and from GitHub comments.

[elyscape]

Just to give a complete picture, this is configurable by the way of assigning SeCreateSymbolicLinkPrivilege.

Weirdly, if your account is part of the Administrators group, you can only create symlinks from an elevated process, even if you also have SeCreateSymbolicLinkPrivilege.

[lygstate]

The most weird part is if we use Guest account, we can create symlink without any permission.

[elyscape]

As a side note, MSYS2 includes native symlink creation with the symlink() function. It also provides native symlink parsing with readlink(), though its signature differs from the POSIX readlink() in that it requires a file handle as input rather than a path.

symlink() will create a native symlink provided that the global variable allow_winsymlinks is set to WSYM_nativestrict at the time the function is called. This can be done by calling set_winsymlinks("nativestrict") or by setting either of the CYGWIN or MSYS environment variables to something containing winsymlinks:nativestrict (these variables are space-separated option[:value] lists).

[dscho]

The most weird part is if we use Guest account, we can create symlink without any permission.

That's an awfully broad statement, given that the same does not hold true here.

As a side note, MSYS2 includes native symlink creation with the symlink() function.

Please note that MSys2's symlink() function creates specially-crafted files _that are only understood by MSys2 programs_.

symlink() will create a native symlink provided that the global variable allow_winsymlinks is set to WSYM_nativestrict at the time the function is called.

The lines below the linked line handle a couple of interesting, but hardly common, filesystems first: NFS (_not_ NTFS!), MVFS, and AFS. Only then does it try to handle _NTFS reparse points_. I was already referring to them in my comment above. NTFS reparse points are similar in concept to Unix-style symlinks, but differ in a couple of quite crucial ways:

• NTFS reparse points only work on... _NTFS_ (i.e. not on FAT or other file systems that may be mapped as network drives or similar),
• you need special privileges to create NTFS reparse points,
• the targets of NTFS reparse points must exist before the reparse points are created,
• NTFS reparse points cannot span file systems, and
• an NTFS reparse point can only refer to a file, or a directory, but the type of the target must not change; if it does, the user must recreate the reparse point.

In particular, something that is quite common on Unix is not possible with reparse points:

ln -s /tmp/blub ~/blob
echo Hello, world. > ~/blob

So you see, the semantics of symlinks and NTFS reparse points are actually quite different when you get to the technical details; And it is the same technical details that actually matter not only in our attempt to emulate symlinks via NTFS reparse points, but even more so they matter in the context of the common operations on symlinks that projects containing symlinks expect to perform.

[elieux]

Weirdly, if your account is part of the Administrators group, you can only create symlinks from an elevated process, even if you also have SeCreateSymbolicLinkPrivilege.

You mean when the Administrators group has this privilege? That's expected behaviour, because UAC essentially does its thing by stripping the Administrators group SID from tokens of un-elevated processes. So if your account has this privilege only by the virtue of being in the Administrators group, only elevated processes get it.

[elieux]

@dscho, are we talking about the same thing? It's true that NTFS symlinks are implemented using reparse points, but they're ostensibly not the same thing, so if a source makes statements about reparse points, it doesn't necessarily mean that the same applies for NTFS symlinks. Specific points below.

NTFS reparse points cannot span file systems

D:\> mklink foo C:\bootmgr
symbolic link created for foo <<===>> C:\bootmgr

D:\> dir foo
29. 04. 2015  01:36    <SYMLINK>      foo [C:\bootmgr]

the targets of NTFS reparse points must exist before the reparse points are created

Although this is true of the Cygwin ln -s implementation (because it needs to know whether to create a directory or file symlink), NTFS symlinks can point to non-existing paths just fine.

D:\> mklink bar C:\bar
symbolic link created for bar <<===>> C:\bar

D:\> type bar
The system cannot find the file specified.

[elieux]

To be clear, the issue of having to specify the type of the symlink (and inconsistencies with how path/to/some/symlink/../more/path is traversed) still makes them unsuitable for general use. (Personally, I don't get why people consider VCS support for symlinks a feature.)

[elyscape]

You mean when the Administrators group has this privilege? That's expected behaviour, because UAC essentially does its thing by stripping the Administrators group SID from tokens of un-elevated processes. So if your account has this privilege only by the virtue of being in the Administrators group, only elevated processes get it.

No, I mean that, if you're in the Adminstrators group but you also have granted SeCreateSymbolicLinkPrivilege to your account specifically, you still need to elevate the process in order to be able to create symlinks. As I said, it's weird.

[elieux]

You're right. That's a horrible feature then (and the permission thing is back on the list of reasons NTFS symlinks can't reasonably be used for git).

Ref: http://stackoverflow.com/questions/15320550/secreatesymboliclinkprivilege-ignored-on-windows-8

[elyscape]

Please note that MSys2's symlink() function creates specially-crafted files _that are only understood by MSys2 programs_.

Unless allow_winsymlinks is set to WFS_nativestrict, in which case it creates native NTFS symlinks.

As for the differences between NTFS symlinks, as implemented via reparse points, and Unix symlinks:

NTFS reparse points only work on... _NTFS_ (i.e. not on FAT or other file systems that may be mapped as network drives or similar),

Well, yes. Presumably, when attempting to check out a symlink, we would have to make sure that the destination supports them, and then fall back to core.symlinks = false behavior if it doesn't.

you need special privileges to create NTFS reparse points,

Also true and another reason why core.symlinks = false should remain the default. That being said, it's not necessarily a reason not to implement support for them, IMO.

the targets of NTFS reparse points must exist before the reparse points are created,

@elieux already addressed this.

NTFS reparse points cannot span file systems, and

@elieux address this as well. In fact, NTFS directory junctions also support spanning file systems, though they're suboptimal for a variety of other reasons (e.g. no relative path support). NTFS hard links don't support spanning file systems, but AFAIK no file system supports that.

an NTFS reparse point can only refer to a file, or a directory, but the type of the target must not change; if it does, the user must recreate the reparse point.

This may not be true. You need to specify a type when you create the symbolic link, but the reparse point data it writes to disk is exactly the same:

I'm looking into how the filesystem determines if a symlink is for a directory or a file. It might be possible to find a way around this issue.

[elyscape]

Okay, never mind on that last bit. While the reparse data is the same, it looks like the other NTFS attributes differ. Specifically, directory symlinks have an $INDEX_ROOT attribute entry and the directory flag set, whereas file symlinks have a $DATA attribute entry and the directory flag cleared.

[dscho]

@elieux thanks for educating me! I have to admit that I only repeated what I read earlier, and that information was clearly wrong, as you proved in a very clear and accurate way.

[dscho]

So, where do we take this issue? Is there a volunteer who needs this feature badly enough to address the issues raised in https://github.com/msysgit/git/pull/231? Or do we need to close this ticket as "WONTFIX"?

[elyscape]

I'm not sure there's a good solution to this when you consider that it's possible to have symlinks in a repository that point outside of the repository or to locations that don't exist and thus don't have a known type.

[kblees]

@dscho @elieux

NTFS reparse points cannot span file systems

More precisely: symlinks by default cannot span _remote_ file systems

> fsutil behavior query SymlinkEvaluation
Die symbolischen Links für lokal zu lokal sind aktiviert.
Die symbolischen Links für lokal zu remote sind aktiviert.
Die symbolischen Links für remote zu lokal sind deaktiviert.
Die symbolischen Links für remote zu remote sind deaktiviert.

[dscho]

... and with English locale

>fsutil behavior query SymlinkEvaluation
Local to local symbolic links are enabled.
Local to remote symbolic links are enabled.
Remote to local symbolic links are disabled.
Remote to remote symbolic links are disabled.

[dandv]

Or do we need to close this ticket as "WONTFIX"?

Closing as WONTFIX would discourage volunteers from even trying.

I'm not sure there's a good solution to this when you consider that it's possible to have symlinks in a repository that point outside of the repository or to locations that don't exist and thus don't have a known type.

In six years of using GitHub I've never seen any other symlink type than the regular one, pointing within the repository. A good solution is better than a perfect solution.

From an end-user's perspective, without knowledge of NTFS internals, if I can run mklink in Windows in the elevated shell in which I run git, I assume git should be able to exec mklink and create the symlink for me.

[PhilipOakley]

So, where do we take this issue? Is there a volunteer who needs this feature badly enough to address the issues raised in msysgit#231? Or do we need to close this ticket as "WONTFIX"?

Hi Dscho,

Given your longer comment, would the right closure be something like "WONTFIX-NEEDSVOLUNTEER", to help readers understand that they could help.

This could be contrasted with those that are at the other extreme "WONTFIX-CONTRARYTOPOLICY" which would advise readers to help elsewhere.

Just a thought.
regards

Philip

[dandv]

Why can't we just leave this issue open?

[dscho]

@dandv Experience shows that "just leaving this issue open" will solve exactly nothing.

So what I need is a volunteer with the need to see this addressed, with enough of the need to dive into it and to work on it. Are you that volunteer?

[dandv]

Just saw the close. Is this fixed? If so, leaving the issue open has helped :-P

[dscho]

Please don't flatter yourself here: @kblees has done all the work. He deserves all of the credit, too.

[dandv]

@dscho: never have I insinuated that I deserve any credit; I don't care for credit. I'm an ordinary end-user who just wants symlinks on Windows, and your insisting on closing an unfixed issue perplexed me. This wasn't a "WONTFIX"; it was a "HELP WANTED". Why would anyone want to close it?

Experience shows that "just leaving this issue open" will solve exactly nothing.

You might want to adjust your priors based on this new experience of leaving an issue open and seeing another dev step in with a fix (thanks @kblees!).

Would he have jumped in if the issue was closed? Who knows. But I have not seen any rational argument for closing this issue, while leaving it open did correlate with it receiving a fix. You may choose to learn from this or not. I have made my case and will no longer engage.

I would, however, like to thank you for all your contributions to Git for Windows.

Best regards,
Dan

[dscho]

@dandv well, as I am revisiting this issue I am left with a little bad taste in my mouth. You could at least have contributed a wiki page describing the intricacies of symlink support on Windows. And let me clarify one thing for you: with Open Source, nobody is an ordinary end-user, because nobody paid for the software. You certainly did not. The downside of getting things for free is that you are expected to do your share to improve the software. You may help others with their problems, you may write documentation about more obscure functions (as -- you guessed it -- symbolic links), you may pay somebody to improve the code. What really, really does not help is to sit back and wait for others to do your bidding.

Oh well, on top of all the work I put into this, I now also have to write that wiki page. _sigh_

[dandv]

Hi @dscho, thanks for looking over this issue again. I'm fully onboard with your thoughts on open source.

As you can see from my GitHub activity, I'm a pretty active contributor and help whenever I can, even with mundane things like fixing typos. I also contribute on Wikipedia, Slant and in many other places, according to my skills and knowledge.

I'm mostly a JavaScript dev these days who uses git on Windows, and those skills and knowledge, unfortunately, don't happen to include Windows API internals or anything about symlinks really, let alone obscure functions. I'm afraid I'd create more work instead in reviewing my naive contributions. As an ordinary user, when it comes to this particular project I can and am happy to contribute with acceptance testing, high-quality bug reporting, proofreading and building awareness of robust solutions (such as with 1, 2).

[dscho]

@dandv okay, half of my rant was trying to prod you to have a look at https://github.com/git-for-windows/git/wiki/Symbolic-Links and edit it :stuck_out_tongue_winking_eye:

[jbuchberger]

Thanks, folks. Are you aware of this git for windows fork supporting symbolic links?
https://github.com/frogonwheels/git/commits/mrg/unicode-symlink-v9

[PhilipOakley]

Thanks, folks. Are you aware of this git for windows fork supporting symbolic links?
https://github.com/frogonwheels/git/commits/mrg/unicode-symlink-v9

I see that the last commits there were 2011, and that they were from Karsten and Michael who has contributed here.

While symlinks, in all their variants, are becoming more common on Windows, I'm not sure if an inclusive solution has been found yet that covers all users.

[dscho]

@jbuchberger we integrated a heavily modified version of what you refer to. Git for Windows does handle symlinks just fine, iff the current user has the privilege to create symlinks, and iff the repository does not say core.symlinks = false. If you clone a repository, and your account has the privilege to create symlinks, Git for Windows will not set core.symlinks, i.e. if you don't play games with user or system config, it will automatically work as desired.

What does not work is to use junction points as if they were symlinks.

[toolness]

As a quick follow-up for anyone who found this thread via Google like me, I think the awesome maintainers of this project just provided what's needed for symlinks on windows to work without admin privileges.

The recent Windows 10 Creators update which was just rolled out does support the creation of symlinks without admin privileges, but it requires that programs like git opt-in to the new behavior by passing a new flag into one of Windows' API calls. This was just merged in #1188, so symlinks should hopefully be a lot easier to deal with in Windows soon! 🎉

[musm]

@toolness thanks for the update. Thus when installing git should we then check use symbolic links to use this feature on new windows versions?