Conan: Symbolic links to directories not preserved when exporting

[atussiot]

Hello,

I'm trying to make a recipe to build Qt5 for macOS and build the modules as frameworks. Everything works perfectly when they are built and used locally. However, if I upload the package to the server, the package is not usable on any machine that will pull this package.

My feeling is that the export of the package is not preserving the symbolic links to directories (which are fundamental in frameworks).

For instance, under QtCore.framework, on the build machine I have:

lrwxr-xr-x  1 atussiot  staff    24B Apr 21 16:06 Headers@ -> Versions/Current/Headers
lrwxr-xr-x  1 atussiot  staff    23B Apr 21 16:06 QtCore@ -> Versions/Current/QtCore
-rw-r--r--  1 atussiot  staff   1.2K Apr 21 16:06 QtCore.prl
lrwxr-xr-x  1 atussiot  staff    26B Apr 21 16:06 Resources@ -> Versions/Current/Resources
drwxr-xr-x  4 atussiot  staff   136B Apr 21 16:06 Versions/

while in the same package downloaded from the server I get (only the link to the binary survives):

lrwxr-xr-x  1 jenkins  staff    23 Apr 20 16:11 QtCore -> Versions/Current/QtCore
-rw-r--r--  1 jenkins  staff  1194 Jan  1  1970 QtCore.prl
drwxr-xr-x  3 jenkins  staff   102 Apr 20 16:11 Versions

This causes CMake to fail in our project:

The imported target "Qt5::Core" references the file "/Users/jenkins/.conan/data/Qt5/5.6.2-0/pix4d/testing/package/7586e5a72612387c5beb630c8ff07f1a397d08b6/lib/QtCore.framework/Headers" but this file does not exist.

I saw that the PR #947 addressed a similar issue. But somehow it doesn't seem to apply to links to directory?

I also checked the conanmanifest.txt and the missing links are not referenced there, perhaps the root cause is that their hashes cannot be computed? Or am I missing something?

Thank you!

[memsharded]

You are very right. Symbolic links right now are only followed for files, not directories.
Not sure if it is a bug, or it has a reason, let's have a look and see if it makes sense to fix it. Annotating it as bug at the moment. Lets try for next release 0.23.

Thanks very much for your detailed report, really useful!

[memsharded]

Hi @atussiot , I have started to have a look at this for next release.
I would like to know more about where the symbolic link is defined, and when it is lost. I thought that it was during the export stage, but now not so sure.

Several possibilities:

• You have some link to a folder in user space, and you are using exports or exports_sources to store the source folder into the recipe
• The link to the folder is defined by the build, and it points to something in the build folder, but when packages are created, just the links are copied into the package folder. This is why it works locally, but later when uploaded it breaks.
  You might be able to inspect the actual files inside your <userhome>/.conan/data folder.

Thanks for your feedback!

[atussiot]

Hi, thanks for taking care of it!

I am not sure these possibilities apply here:

• We are using exports, but only for a trivial patch.
• The structure of frameworks on macOS is relatively straightforward and self contained (see the trees in the documentation). The links are relative inside the framework so I don't think the problem is in the location of the links' targets. I guess they are created during the build or install with make.

I tried removing everything under ~/.conan/data, except the package and it still works fine locally. To me it really looks like it is something happening during the upload.

Let me know if there is something else I can test/check/do to help!

[memsharded]

I am working on this, and my initial tests are confusing. It is true that symlinks are not maintained while packaging, from the "build" to the "package" folder, with the package() method.
When symlinks are defined in the build between folders, such folders and files are actually duplicated in the package folder, so everything keep workings, the only problem is that files are duplicated wasting extra disk space.
So I would like to know a couple of things:

• The layout you showed me above was in the "build" or in the "package" folder?
• Can you also have a look to the other folder, and see how it looks like in it?
• Can you share with me also your package() method.
  Thank you very much!

[atussiot]

Hmm interesting. I am actually not using the package() method at all. I simply use the make install command that copies everything directly into the package folder from the build() method (hence also duplicating). Something like this:

    # Tell the configure script to install directly in the package folder
    self.run('cd {} && ./configure -prefix {}'.format(self.project_root, self.package_folder))

    # Frameworks are created here (with the symlinks within themselves)...
    self.run('cd {} && make -j {}'.format(self.project_root, cpu_count()))

    # ... and copied here from build to package folder.
    self.run('cd {} && make install'.format(self.project_root))

The layout I showed above was in the package folder, and it is the same in the build one.

In the duplication you describe, you mention both folders and files. I guess the problem I'm having is different as the links pointing to files are preserved. Wouldn't they be lost too if it would be a matter of where the files/folders are located?

[memsharded]

Good to know. Then, I think the issue has two different parts: one would be to preserve folder links in the package() method, because eventually someone else will suffer this problem, even if you are fine now with make install, and then also check the compression and decompression of the package .tgz, to check what is happening.

It is ok that symlinks to files are preserved, because they are explicitely handled. The duplication only happens if the symlink is done between folders (because symlinks=True, so directory walk will traverse it, but then files will not be symlinks and be treated as different files).

I will have another look, thanks for the info!

[atussiot]

Agreed, I guess it is important to look at both. In fact, I am facing other issues with folder because the copy() method doesn't let me copy them. I'm still not sure how to properly handle it in client conan files. It would be handy to be able to do something like self.copy('*.framework', 'lib', 'lib', symlinks=True). But I guess that should be reported as a separate issue?

But regarding my current issue, I believe checking the compression and decompression of the package is the way to go. Let me know if I can help with that too!

Out of curiosity, would you know maybe if there are examples of anybody attempting to handle frameworks like this before?

Thanks again for handling it!

[memsharded]

I would like some others feedback on this issue.
The problem that I am facing to implement this is that it might break existing packages, because the generated tgz will be different, zipping folders symlink that previously were not being packaged.

So I would like to understand the real goal of such symlinks. They seem to make sense to me (from my ignorance of OSX frameworks) when there are different versions, so the symlink between "latest"->"versions/2" is useful. But conan already provides the solution for handling different versions, without symlinks. Packages will work fine just by packaging from the folder you want.

So please, I would like to know:

• Is it a consumer issue? Is the consumer of the package expecting to find a "Headers" symlink pointing to somewhere else? Is it good to have directly a Headers folder?
• Is someone already using this approach, which might end with a different tgz and different installed package with symlinks if we change the way the tgz is created to maintain symlinks? cc/ @tru

I keep working on it, trying to find a way. Thanks for the feedback!

[memsharded]

I have submitted the PR anyway, in case someone wants to check it. https://github.com/conan-io/conan/pull/1289

[memsharded]

The PR has been merged, so it will be in next release :)

[atussiot]

Great news, thank you very much! Looking forward to testing it!

[memsharded]

Thanks! If you haven't yet, you probably want to subscribe to the release announcement mailing list (in conan.io footer). Exclusively for announcing new releases. We will be announcing the release candidate and release there.

[memsharded]

Hi @atussiot , this is already released in 0.23, could you please upgrade and try it? Thanks!

[atussiot]

Hi @memsharded!

Sorry for the late reply, I will give it a try today and let you know how it goes!

[atussiot]

As far as it concerns the reported issue, it now works as expected, thank you very much again!

However I run into a new problem, in the conanfile.py of the test_package. The only way I figured out so far to copy the folders is:

def imports(self):
    self.copy("lib*", root_package="Qt5")

But apparently this one is not preserving links (not even for binaries this time), leading to a crash. Is that expected or am I doing something wrong?

[tru]

self.copy() takes a links argument. Set that to true and that should work.

[memsharded]

Also, there is an alternative approach you might be interested in having a look. Instead of copying the shared libraries (imports is mainly for that) from the Qt package folder, to the test folder, you can try making the Qt package to add itself, or its bin or lib subfolder to the path, or to LD_LIBRARY_PATH, depending on the system. Something like (not tested):

def package_info(self):
    if self.settings.os == "Windows":
        self.env_info.PATH.append("bin")
    elif self.settings.os == "Linux":
       self.env.info.LD_LIBRARY_PATH.append("lib")
    else: ...