Conan: [bug] Short paths and cygwin

[Daniel-Roberts-Bose]

I apologize if this issue has already been raised before but I searched and couldnt find the exact same issue I am having.

The short version is that I am running on windows, inside a cygwin shell as is required by a third party vendor that we use for critical software (aka we cant change the directory structure of the code, the compilers or tools they are requiring). I am running into an issue in conan where when I build in cache via a conan create command, it errors out because it reaches the path limit on windows of 260 characters.

I have tried using the conan short paths variable in both my conanfile as well as setting the global configuration to use CONAN_USER_HOME_SHORT and CONAN_USE_ALWAYS_SHORT_PATHS but neither seem to be working.

My thoughts are that since I am running inside a cygwin shell, conan thinks the environment is unix and does not apply short path logic. Any ideas on how to try debug this?

Environment Details (include every applicable attribute)

• Operating System+version: Windows 10 + cygwin
• Conan version: 1.20.5
• Python version: 3.6

Steps to reproduce (Include if Applicable)

• Use cygwin
• Run conan create using short_paths = True, and a very long directory structure

Logs (Executed commands with output) (Include/Attach if Applicable)

[jgsogo]

Hi, @Daniel-Roberts-Bose

Current implementation only considers short_paths if platform.system() return Windows https://github.com/conan-io/conan/blob/73b8734bdb404d8ff0cd5521e829513c77501ff7/conans/paths/package_layouts/package_cache_layout.py#L26

I've just installed Cygwin in one machine, it comes by default with Python 3.5.2 and it returns "Windows". Is it different in your cygwin?

[Daniel-Roberts-Bose]

@jgsogo that call in my case does not return windows. It instead returns the following:

>>> platform.system()
'CYGWIN_NT-10.0-17763'

So this explains why short paths are not working for me

I should point out that I am not using windows native python, I am using the one that gets installed within cygwin

[Daniel-Roberts-Bose]

As another side note, is there any reason we cant allow short paths on ANY system? I understand that the general use case is only for windows but what is the down side of allowing this on say linux or mac os?

[Daniel-Roberts-Bose]

Perhaps some additional useful info

$ python3
Python 3.6.9 (default, Jul 21 2019, 14:33:59)
[GCC 7.4.0] on cygwin

[jgsogo]

Yes, the change is easy to implement. I'm not aware of any reason to avoid short_paths for other systems than Windows, but at least we need to enable it for Cygwin (easy check).

[memsharded]

As another side note, is there any reason we cant allow short paths on ANY system? I understand that the general use case is only for windows but what is the down side of allowing this on say linux or mac os?

The short_paths feature is way more fragile and problematic than the normal cache. It is a total nightmare (maintenance, bugs, inconsistencies and hacks in our codebase) of a feature, and we provide it because there is no other alternative for Windows. So we want to limit it to the minimum, no way we are enabling it for other platforms 😅

[Daniel-Roberts-Bose]

Thats fair but enabling it at least for "windows" platforms such as cygwin or minGW would be good :)

[memsharded]

I am having a look to this, and I have a concern. There are 2 very different scenarios:

• Using the terminal and command line as utilities to run things, and build, but actually using Visual studio, CMake and other things in the native Windows environment
• Using Cygwin GCC compiler and the Cygwin POSIX environment (cygwin dll). This binaries are probably incompatible with native Windows ones (even if built with MinGW gcc, because the POSIX cygwin dll)

I would like to understand better how the community is using Cygwin, which cases, etc. Please feedback cc/ @jokram @tru @sztomi

[Daniel-Roberts-Bose]

I can only speak for myself but the majority of compilations I am doing are cross compiles to some custom architectures.

If I understand your concern correctly, then you're suggesting or saying that something compiled with native cygwin and windows GCC compilers, will only run on their given platform but again, that's assuming things are not cross compiled.

I don't personally see how this would be too much of a concern because it's likely you are either using one or the other and not trying to mix them but, if you do have to compile for one or the other could cross compiling still work fine?

[sztomi]

Using the terminal and command line as utilities to run things, and build, but actually using Visual studio, CMake and other things in the native Windows environment

This only works if the build tools are aware that they are running in Cygwin under Windows, or if they are specifically built for cygwin. For example, the build of ICU is an Autotools build that is capable of building under cygwin with CC=cl.exe. But some other tools/builds fail usually because of the filesystem differences (/cygdrive/c/... vs. C:\... and the symlink implementation of cygwin) or the terminal itself etc. So if you are using CMake, you need to make sure that you are running a cmake.exe that is built for Cygwin (this is what we do, e.g. instead of our custom built CMake, we use the one from the Cygwin package).

Using Cygwin GCC compiler and the Cygwin POSIX environment (cygwin dll). This binaries are probably incompatible with native Windows ones

I think that depends on what we are talking about when we say binaries. I assume you meant either shared libraries or static ones (executables are obviously native, because that's the basic premise of Cygwin).

Static libraries are most certainly incompatible, except maybe in rare cases where your program does not refer to the POSIX layer (i.e. the linker doesn't need to find symbols that don't exist on native windows). In this rare case, a native windows ld might be able to link your static lib, but this is definitely a niche use-case and requires a lot of care.

For shared libraries, I think it's easier to make it work, if the calling convention is agreed upon. A C interface can be called in the DLL, even if it's a Cygwin-built one. It still needs cygwin-1.dll, and maybe others, but in terms of binary compatibility, this is viable.

But both of these assume that your library is aware that it might be used under native Windows, so is able to adapt to the filesystem challenges. Again, this makes it a niche setup. So if the point of this discussion is to learn what should be added to the default settings, I think it's best to consider it incompatible. If someone really wants to make it work, let them use their own settings.

(even if built with MinGW gcc, because the POSIX cygwin dll)

If you are cross-compiling your program for Windows with a mingw-gcc that is running under cygwin, you won't be depending on the Cygwin POSIX layer, so the binaries do not need cygwin-1.dll (you also _can't_ depend on it anymore). This is no different than cross-compiling from Linux to Windows with mingw-gcc, except mingw-gcc itself takes advantage of the POSIX layer to run under Windows.

[jokram]

I think some of our projects are using this one:

Using the terminal and command line as utilities to run things, and build, but actually using Visual studio, CMake and other things in the native Windows environment

In some of the projects cygwin is used for 2 main reasons:

• The “official” git-repo (https://gerrit.googlesource.com/git-repo/) version had a dependency with a python library existing only in the Unix version of python (and therefore the python version available in Cygwin). We know that this is not valid anymore and have already changed that.
• It is extensible compared to Git Bash: you can deploy a ready-to-use Cygwin package which contains all the libraries and software you want the users to have.

In Cygwin they use the commandline utilites, but cross-compile for embedded software. And the cross-compiler is Windows native.

[jgsogo]

Hello, Cygwin users, how do you deal with Cygwin version? Do you encourage your users to have always the latest cygwin available? Do you maintain yourselves a fixed version for all the company and compatible packages for it in your own server? Here we see differences between several Cygwin versions (https://github.com/conan-io/conan/issues/6662), this info will be useful to make up our mind and see the best way to work with Conan+Cygwin. Thanks!

[Daniel-Roberts-Bose]

We have a set version we tell everyone at our company to download, however we don't prohibit people using whatever version they want to and we don't have a way to enforce versions on our devs. We haven't seen any real problems with using different versions of cygwin and packages. All the major and important packages people need are installed via a script we provide, but that list only has maybe half a dozen packages on it like python and pip

[sztomi]

@Daniel-Roberts-Bose I might be mistaken, but I the cygwin version will not be representative of the versions of packages installed in the system. Furthermore, Cygwin mirrors don't keep many versions of packages around.

@jgsogo For the above reason, we have a zipped cygwin installation (which is created by a python script). This is distributed to build nodes via chef. Our developers don't need cygwin locally, because it builds just one package (openssh) which is an "auxiliary binary" from our point of view. We build its dep tree statically, so our openssh package is just a static binary plus two cygwin DLLs that can't be linked statically. Nothing links to our openssh package via a linker. If our devs did need cygwin locally, I would tell them to grab the same zip.

[jgsogo]

Hi @Daniel-Roberts-Bose

I'm implementing the short_paths for Cygwin in #6741, there are different scenarios here:

1. Using _Cygwin - Conan_ inside Cygwin (installed with the Cygwin python):

   • By default Conan uses home as /home/<user> and will put the short_home in the user home like /home/<user>/.conan_short. 💚
   • If the user is forcing Conan to use the same cache as the Windows one, the user needs to set CONAN_USER_HOME=/cygdrive/c/Users/<user> and Conan will use the same short_home as the Windows one: /cygdrive/c/.conan. 💚
   • If using CONAN_USER_HOME=C:/Users/<user>, Conan fails (see note (1) below) 🔴

2. Using _Windows - Conan_ inside Cygwin (installed with the _native_ Windows python):

   • By default, Conan will use C:\cygwin64\home\<user>\.conan as Conan home (see note (3) below) and Conan will place the short_paths into C:\.conan 💛
   • If using CONAN_USER_HOME=/cygdrive/c/Users/<user>, it fails (see note (2) below) 🔴
   • If using CONAN_USER_HOME=C:/Users/<user> it works, Conan will use C:/.conan for short_paths 💚

Using CONAN_USER_HOME_SHORT environment variable, Conan skips a lot of logic, we can expect bugs/issues associated with it, probably we cannot cover all the casuistics.

---

Note.-

1. Conan-Cygwin running in Cygwin: using CONAN_USER_HOME=C:/Users/<user> fails because Conan requires an absolute path and it isn't from the unix perspective.

1. Conan-Windows running in Cygwin: using CONAN_USER_HOME=/cygdrive/c/Users/<user> fails because Conan will compute the home like C:\cygdrive\c\Users\<user>\.conan and it is not a valid path.

2. Conan-Windows running in Cygwin: it will detect the Conan user home like C:\cygwin64\home\<user>\.conan which is /home/<user>. Is it expected? Should it return the path to C:/Users/<user>?
   IMO, if it uses the Unix-home, it should use the /home/<user/.conan_short too. But, I'd prefer it if Conan uses the Windows-home.

I'll open an issue with this.

[Daniel-Roberts-Bose]

This is great! Thanks for the hard work and effort! :)

[sztomi]

@jgsogo Native windows Python running inside Cygwin is going to have a lot more problems, I'm not sure if it's worth (or really possible) supporting that scenario. All os.path functionality will be broken, os.pathsep will be \ (which is wrong for cygwin), symlinks won't work etc.

[jgsogo]

I've been able to run successfully several recipes using _native windows Python_ inside Cygwin and, given some of the issues that are reported, I feel like there is people using it this way. But yes, I agree with you, there will be more issues and probably claiming that this scenario is supported by Conan will be impossible. We need to think about it and decide if we say nothing or we want to detect this scenario and print a big warning.

BTW, my Cygwin works with \ in paths inside native-Windows-Python (not in the terminal)

[sztomi]

BTW, my Cygwin works with \ in paths inside native-Windows-Python

No, it is your native python that works with \ paths, because only binaries that are compiled with Cygwin will call the cygwin emulation layer. Your native python is just a normal binary. Issues happen when the two worlds interact, like your native python (assuming C:) interacting with something that assumes /cygdrive/c. Or symlinks etc.

[jgsogo]

I promise I can't remember what I mean with _"BTW, my Cygwin works with \ in paths inside native-Windows-Python (not in the terminal)"_, now I think it was a typo and I was thinking about Cygwin-Python, but I can't remember 😓 (anyway my Cygwin-Python works with Windows-style paths, although it is strongly discouraged).

Mixing both worlds will be a nightmare, but I'm sure that many people are using Conan-Windows inside Cygwin just because "it works"... until it fails. We need to identify these situations and handle them better (maybe even raise if the user is mixing both worlds).

[sztomi]

Agreed 100%