Conan: Conan 0.28.x : conan install points to wrong directory for packages previously built with short_paths=True

[jcar87]

Windows 10, Python 3.6, Conan 0.28.x

If we call "conan install packagename -g cmake", on a package built and packaged on an earlier version of conan, the cmake variables will no longer point to C:/.conan/xxxx but to the actual cache location (and the files are not there, obviously).

What would be a way to proceed?

[memsharded]

Yes, from Conan 0.28.x, if you have Windows 10, with Python 3.6 and LongPaths windows registry key (SYSTEM\CurrentControlSet\Control\FileSystem->LongPathsEnabled activated, you do not longer need to manage short paths. Windows already supports large paths. So the short_paths feature is no longer necessary, and it is a no-op.

Are you having problems with this new behavior? Maybe @tru or @sztomi can provide more feedback. Should we provide a way to for short_paths again, even if the windows registry key is activated?

[sztomi]

Oh yes, it should be allowed still, despite long paths being enabled in the registry. The reason is that while in the current state conan supports long paths very well, not all build systems do, so they may still refuse or choke on long paths. And usually they are precisely the worst offenders, like Boost. Sometimes it's just a mysterious error, probably a buffer overflow. We ended up with short_paths set to True after fighting a lot with them. I expect it will take quite some years for software to support long paths on windows. There is also a lot of confusion about it because people will think \\?\-style UNC paths when "long paths" and "Windows" are mentioned in the same sentence.

[jcar87]

The problem I'm currently having is the following.
For a package with short_paths=True retrieved from a remote while 0.27.0 was installed, the package was placed by conan in c:\.conan\zka3gc3o\1.
Running conan install with the cmake generator and that package, will have the cmake variable CONAN_XXXX_ROOT (defined in conanbuildinfo.cmake) point to that directory.

After installing conan 0.28, however, running conan install will have CONAN_XXXX_ROOT point to C:\users\username\.conan\data\XXXX\version\user\channel\package\hash\. The only thing in there is a .conan_link file.

[memsharded]

Oh, that sounds indeed as a bug. Are you sure you completely wiped the local cache, just in case it had something inconsistent? Annotating as a bug, I have to check it. Thanks for telling!

[memsharded]

@jcar87 I have been trying this, and it seems working fine. I would say that you have to remove your local cache first, it is not possible to upgrade the "short_paths" feature, without doing it, as all the cache is assuming such short_paths.

I am going to add an opt-in to use short_paths even if in Win10, and LongPathsEnabled at the system level.

[jcar87]

I have able to reproduce this, but not through that registry key.

launch gpedit.msc and navigate to
Local Computer Policy > Computer Configuration > Administrative Templates > System > Filesystem > NTFS.
The setting is called "enable win32 long paths"
Not sure the registry key is used anymore according to this: https://superuser.com/questions/1119883/windows-10-enable-ntfs-long-paths-policy-option-missing

If it is enabled and _the package was installed prior to enabling it (or by an earlier conan version)_, then conan install will produce a conanbuildinfo.cmake pointing to the directory in %userprofile%/.conan/data . If it is disabled, then conan install points to c:\.conan\...

I can confirm that deleting the package and fetching it again from the remote while long paths are enabled, solves the problem.

Is it the common M.O. having to clear the local cache every time conan is upgraded to a newer version?

The strange thing is that the computer which was having this problem, didnt have long path support enabled.

[memsharded]

Thanks for the info!

If it is enabled and the package was installed prior to enabling it (or by an earlier conan version)

Yes, that is exactly the point. In general, it is not necessary to clear the local cache when upgrading. But this release (we are still in beta 0.X) has introduced a few breaking changes (this is one): http://docs.conan.io/en/latest/changelog.html, so in this case, yes, it is recommended to clear the cache

You can check the implementation in: https://github.com/conan-io/conan/blob/develop/conans/paths.py#L10

Maybe some of the other Win10 configurations using a different registry key?

[memsharded]

I was about to add the opt-out (i.e. to really use short_paths even if python 3.6 and LongPathsEnabled at system level), but I have some doubts:

• @sztomi says that when py36 and OS LongPathsEnabled are active, the failures comes from limitations of build systems and other apps, which seems difficult to debug and know what is happening.
• If we make it opt-out, those difficult to debug errors will happen anyway
• If we change it, and make using Windows long-paths an opt-in then we will break again (necessary to clean the cache, only that), and basically the code checking that long-paths are enabled is unnecessary, as it is finally something defined by the user in an opt-in variable

What do you think? cc/ @lasote

[sztomi]

If we change it, and make using Windows long-paths an opt-in then we will break again (necessary to clean the cache, only that), and basically the code checking that long-paths are enabled is unnecessary, as it is finally something defined by the user in an opt-in variable

Yeah, so this is an unfortunate situation... I have to change my stance on this because now I think opt-in is better. Unfortunately, when we were discussing it earlier I thought that after enabling long paths in Windows everything will just work. And if that were true, conan would be right to automatically use the long paths feature if it's available. But build tools break, and some of the worst offenders in path length are also the ones that break the most (Boost, for example). After fighting with this a lot, I came to the conclusion that it's going to take a long-long time until everything starts to support long paths properly.

I would say that the checking code can still be salvaged: users could be prompted with a link to learn about long paths (a page in the docs) and enable it if they want to.

[memsharded]

I would say that the checking code can still be salvaged: users could be prompted with a link to learn about long paths (a page in the docs) and enable it if they want to.

No prompts, that would halt many CI processes, etc.
If opt-in is better, better do it now than later. Thanks for the feedback!

[sztomi]

No, I didn't mean an interactive prompt, of course :)

[lasote]

Agree, better to step back and make it opt-in if it's not solid enough. Thanks all for the feedback.

[memsharded]

CONAN_USER_HOME_SHORT=None has been recovered as the way to opt-out from using short_paths feature in windows.