Conan: Linux package upload makes Windows one outdated from recipe

[db4]

I have a package that is built by Gitlab CI for Linux/Windows and then uploaded to an internal Artifactory Community Edition server (6.0.1 if it matters). The problem is that when the Linux package is uploaded , the Windows one becomes outdated from recipe and vice versa. Looks like it's due to different line endings in conanfile.py (Windows CI uploads it with CRLF while Linux CI with only LF). Is it a bug or just some misconfiguration?

[memsharded]

No, conan does not change any line ending at all. You need to make sure that when you checkout the recipe sources, you are consistent with line endings. I'd suggest going LF (Linux) always, in all platforms, specially now that it is supported by all editors.

[db4]

Thanks for the explanation. I've set core.autocrlf=input for Windows Git and now all problems are gone. But I still think that EOL normalization of the recipe by conan on export (like Git does) would be quite useful.

[memsharded]

Good to know!
The thing is that normalization is not as easy as it seems, for example for files without extensions. And it is always possible that you normalize something that the users really don't want to normalize for some valid reasons, breaking and blocking. The git implementation for that is not trivial, and it has some configuration dedicated to it. Not to say that this would further slow the operation in Windows, that is already the slowest one.

So we have considered it a few times, and we have always stepped back (also from bad experiences in the past with basically the same issue). Nowadays, I am a believer that all text files should normalize to LF, there is no reason to have different formats, as having such different formats will sooner or later bite you (like this time with conan, but also with git or any other SCM system that is misconfigured or fails to normalize, and then you have a million lines diff :) )

[db4]

The thing is that normalization is not as easy as it seems, for example for files without extensions.

When conan detects if a package is outdated from recipe, only conanfile.py is involved, right? Why not to always normalize conanfile.py to LF on export?

I am a believer that all text files should normalize to LF

I agree absolutely. The problem is that CRLF (core.autocrlf=true) is default for Windows git. In my case I had to rebuild two Windows docker images with git config --system core.autocrlf input. It is not always possible if you use prebuilt CI infrastructure.

[lasote]

When conan detects if a package is outdated from recipe, only conanfile.py is involved, right? Why
not to always normalize conanfile.py to LF on export?

No, any exported file is checked also.

[memsharded]

Exactly, the problem is not the conanfile.py, but other possible files that could be in exports or exports_sources. Files without extensions or files that even with a recognizable extension, modifications are not desired (some data files, for example)

[danimtb]

I have opened and issue in the docs repo to include this as an answer (probably in FAQ section) conan-io/docs#693

[tonka3000]

Hey @memsharded ,

I have the same problem from time to time, and i mostly solve it via „all to LF“, but I think it would be great to have an opt-in feature to control the hashing of the exported files including conanfile.py . If this feature is turned on (maybe an definition of an varible is enough), the conanfile writer has to define exatcly which file (fnmatch filter)he/she wants to be independent of the lineending.

Example:
...
name = „test“
exports_sources_ignore_lineending = „myfile.py“, „.txt“

Greetings

Tonka

[liberforce]

I also had this problem and I agree with @memsharded : the solution here isn't in conan. However Conan could provide some best practices for this common use case.

I have tried changing the core.autocrlf to false to force using the same end of line on both platforms, so that the recipe doesn't change, but this is a bad idea as it doesn't mix well with continuous integration. I'm using gitlab-ci, and the clone is done _before_ I can change core.autocrlf. I could apply the setting globally on the gitlab runner creation, but I don't want to, because the runner may be shared by different projects that don't necessary want core.autocrlf to be false.

So I tried changing the setting only for the specific clones that need it (the ones that have conan recipes) like this:

git config core.autocrlf false
git checkout .
git submodule foreach "git config core.autocrlf false"
git submodule foreach "git checkout ."

This kind of works, but I had a problem that took me time to find. My gitlab-ci script calls a bash script on Linux and a .bat script on Windows to do some setup, and that's where I was calling the commands above. The problem is that git checkout . was modifying the .bat file while it was being executed, which was leading to strange errors. It also seems problems might occur on .bat files using LF instead of CRLF.

So, I had to use a .gitattributes file to tell that .bat files should always use CRLF. And if I have to use a .gitattributes file anyway, then setting there .py file to use LF will solve 90% of the problems and won't require post-clone hacks. I was reluctant to use that solution, because it doesn't handle submodules, but this should be inside every module that cares about being cross-platform, really.

So I think the best fix is to have at least this in a .gitattributes of every repository that contains conan recipes:

*.bat text eol=crlf
*.py  text eol=lf

If that's not enough because of potential additionnal files exported (.patch/.diff files), then fixes should go inside the .gitattributes file anyway.

My 2¢...

[danimtb]

would you mind adding your solution to https://docs.conan.io/en/latest/faq/using.html#packages-got-outdated-when-uploading-an-unchanged-recipe-from-a-different-machine @liberforce? That would be very helpful for other users! 😀

[liberforce]

Thanks for the link. I'm still trying to get things sorted out, as there are different use cases to consider. If I get anywhere near a final solution, I'll propose it.

[atrelinski]

Could You update https://docs.conan.io/en/latest/faq/using.html#packages-got-outdated-when-uploading-an-unchanged-recipe-from-a-different-machine by listing which files impact recipe hash?

I've added "conanfile.py text eol=lf" in .gitattributes and the same for all other files listed in "exports" and "exports_sources" in conanfile.py and for all files that may impact any of them. Unfortunately that didn't help.

Finally I had to change to "* text eol=lf" which is very not nice solution.

I really appreciate your help in that matter

[memsharded]

Hi @atrelinski

There is not fixed list of files that are exported. I think the best is if you inspect the conanmanifest.txt that you will find inside the package (you can see it navigating into the .conan cache, or in Artifactory server if you uploaded it). That file contains all the files that were exported together with their checksum. The hash of the recipe is the hash of that file (without the timestamp). Please tell me if this helps.

[atrelinski]

Hi @memsharded
Thanks for Your help.
I confirm that helps.
LICENSE file was not in "exports" neither "exports_sources", but created on the fly in package() method.
Indeed it was mentioned in conanmanifest.txt which helps very much.

Could You update https://docs.conan.io/en/latest/faq/using.html#packages-got-outdated-when-uploading-an-unchanged-recipe-from-a-different-machine that recipe hash is calculated from files listed in conanmanifest.txt and outdating can be easily prevented by listing those files in .gitattributes by adding " text eol=lf" for each file?

[memsharded]

Hi @atrelinski

Glad that it clarified something.
I have submitted PR https://github.com/conan-io/docs/pull/1473, if you want to have a look and see if it will help to other users. I have added also the global .git/config solution as well, I think it is simpler and more robust to changes/additions in recipes.

[atrelinski]

@memsharded
Change is perfect!
Really professional when additionally proposing solution for the problem!
Thank You

[memsharded]

Closed by https://github.com/conan-io/docs/pull/1473