Conan: [question] Why is the full path of the CI generated package in my buildinfo?

[cajus]

I've a project based on the qmake generator which provides a conanbuildinfo.pri file. The project uses a conanfile.py for it's dependencies and gets ready using conan install ....

The dependencies are basically built in the CI using combinations of create and upload, they are built as the user dev which resides in /home/dev. Example:

$ conan create --profile someprofile -s compiler=gcc -s compiler.version=8 . foo/stable
$ conan upload -r repo --all --force qt/5.13.2@foo/stable

When I retrieve the project dependencies via a top level conan install ..., the paths for the static libraries used in the qt package are not adapted and still reference the CI's /home/dev/... - which in turn do not exist locally.

To work around this, I've to install the Qt package manually before any other conan install run like this:

$ conan install --profile someprofile qt/5.13.2@foo/stable
$ conan install --profile someprofile .

Is there a way to get this get around this?

• [x] I've read the CONTRIBUTING guide.

[memsharded]

Hi @cajus

I think this might be an issue of the qt package. The bincrafters version is using the packaged Config.cmake file, and it might have issues, like absolute paths. But it shouldn't affect if you are using the qmake generator.

But I am missing some info, where do you get that full path? in the generated conanbuildinfo.pri? Are those packages somewhere we can check or are they private? If private, there is some reduced reproducible case that we could try? Thanks!

[cajus]

Hi @memsharded,
thanks for the quick answer. In fact I can't find any reference to /home/dev inside the conanbuildinfo.pri file. Doing a lazy rgrep in ~/.conan/data shows up references like these:

qt/5.13.2/foo/stable/package/bdb78385358930deb9be260025e3bdf925800113/plugins/platforms/libqminimal.prl:QMAKE_PRL_BUILD_DIR = /home/dev/.conan/data/qt/5.13.2/foo/stable/build/bdb78385358930deb9be260025e3bdf925800113/qtbase/src/plugins/platforms/minimal
qt/5.13.2/foo/stable/package/bdb78385358930deb9be260025e3bdf925800113/plugins/platforms/libqminimal.prl:QMAKE_PRL_LIBS = $$[QT_INSTALL_LIBS]/libQt5EventDispatcherSupport.a $$[QT_INSTALL_LIBS]/libQt5FontDatabaseSupport.a $$[QT_INSTALL_LIBS]/libQt5Gui.a /home/dev/.conan/data/harfbuzz/2.6.4/foo/stable/package/c68551ae35bf5d62e66263379d58a38416eb84a9/lib/libharfbuzz.a /home/dev/.conan/data/freetype/2.10.1/foo/stable/package/387eb5152986b9b3cbc2ebb94607d96d90674d67/lib/libfreetype.a /home/dev/.conan/data/libpng/1.6.37/foo/stable/package/f99afdbf2a1cc98ba2029817b35103455b6a9b77/lib/libpng16.a /usr/lib64/libm.so $$[QT_INSTALL_LIBS]/libQt5Core.a -lm /home/dev/.conan/data/pcre2/10.32/foo/stable/package/18903774d26ee0498535ef95198a1c997e4ca9ba/lib/libpcre2-posix.a /home/dev/.conan/data/pcre2/10.32/foo/stable/package/18903774d26ee0498535ef95198a1c997e4ca9ba/lib/libpcre2-8.a /home/dev/.conan/data/pcre2/10.32/foo/stable/package/18903774d26ee0498535ef95198a1c997e4ca9ba/lib/libpcre2-16.a /home/dev/.conan/data/pcre2/10.32/foo/stable/package/18903774d26ee0498535ef95198a1c997e4ca9ba/lib/libpcre2-32.a /home/dev/.conan/data/zlib/1.2.11/foo/stable/package/6af9cc7cb931c5ad942174fd7838eb655717c709/lib/libz.a /home/dev/.conan/data/bzip2/1.0.8/foo/stable/package/da606cf731e334010b0bf6e85a2a6f891b9f36b0/lib/libbz2.a /home/dev/.conan/data/double-conversion/3.1.4/foo/stable/package/b911f48570f9bb2902d9e83b2b9ebf9d376c8c56/lib/libdouble-conversion.a /usr/lib64/libdl.so /usr/lib64/libGL.so -lpthread /home/dev/.conan/data/freetype/2.10.1/foo/stable/package/387eb5152986b9b3cbc2ebb94607d96d90674d67/lib/libfreetype.a /home/dev/.conan/data/libpng/1.6.37/foo/stable/package/f99afdbf2a1cc98ba2029817b35103455b6a9b77/lib/libpng16.a /usr/lib64/libm.so /home/dev/.conan/data/bzip2/1.0.8/foo/stable/package/da606cf731e334010b0bf6e85a2a6f891b9f36b0/lib/libbz2.a /home/dev/.conan/data/zlib/1.2.11/foo/stable/package/6af9cc7cb931c5ad942174fd7838eb655717c709/lib/libz.a
...
qt/5.13.2/bje/stable/export/conanfile.py:            paths_to_replace = ["/home/dev/.conan"]

Qt has been built statically. It _seems_ like the replacement of /home/dev/.conan only happens when qt is installed explicitly by a single conan install ... qt/5.13.2@foo/stable call. And that's true for all it's direct dependencies that are not directly listed in the main conanfile.py.

It looks like my qt/conanfile.py differs from the bincrafters/conan-qt one: the deploy method is gone. Let me sync it and try again...

[cajus]

Ok. While syncing the two files, I rememberd that we added this to the Qt conanfile.py, because the CI build never worked for us:

   def deploy(self):
       if self.settings.os != "Windows" and self.settings.os != "WindowsStore":
           # Because Qt gets compiled with apths to dependencies compiled into a smattering
           # of files, we'll have to find these paths and replace them with the correct path
           # for the user building the application.
           print("Running post-installation hook to fix dependendy paths...", end="")
           user_conan_home = os.path.expanduser("~") + "/.conan"
           paths_to_replace = ["/home/dev/.conan"]
           extensions = (".h", ".prl", ".pri", ".pc", ".la")
           files_to_fix = []
           for root, dirs, files in os.walk(self.package_folder):
               files_to_fix += [os.path.join(root, f) for f in files if f.endswith(extensions)]
           for f in files_to_fix:
               for p in paths_to_replace:
                   tools.replace_in_file(f, p, user_conan_home, strict=False)
           print("done")

That explains why it does work when calling conan install qt.... deploy() isn't called otherwise.

Let me refine the question: what needs to be done to make the Qt package fix it's files when being installed as a dependency?

[memsharded]

Ok, now I see what was your issue, thanks.

Let me refine the question: what needs to be done to make the Qt package fix it's files when being installed as a dependency?

Right now it is not possible. The packages are "immutable", they have a conanmanifest.txt file that declares the files inside the package with their checksum. Any action that modifies those contents will end in a "corrupted" package, and that can be problematic.

We have been considering to add to recipes an extra "post-install" step that would work doing a copy of the package contents, and allowing modifying it, but this is complicated, and not yet in the roadmap. We might consider it in the future. And probably even in that case, this should be used just for exceptional cases, the default should be that packages are immutable and relocatable from their origin, no need to modify them locally.

I think in the meantime, the way to go is to try to make the qt recipe to generate relocatable packages, no package should contain absolute paths. There are helpers like cmake.patch_config_paths(), maybe something in that line, applied in the recipe before packaging it?

[cajus]

Hmm. It's a bit irritating that even the upstream installed Qt seems to patch the la/pc/etc. files to include the users home... But as qmake generates just Makefiles, I might be able to put some placeholders in the affected files. Will try it tomorrow.

[cajus]

That did last some minutes now. It seems that rewriting the fixed home to ${HOME} before packaging resolves the issue. That's not a nice solution, though. As long as you don't use CONAN_USER_HOME it works.

[memsharded]

That did last some minutes now. It seems that rewriting the fixed home to ${HOME} before packaging resolves the issue. That's not a nice solution, though. As long as you don't use CONAN_USER_HOME it works.

And wouldn't a replacement to CONAN_USER_HOME instead work, even if you use that variable? Maybe that is a bit more robust?

[cajus]

Is it always set (i.e. to $HOME) when running make? I can't see it anywhere in the conanbuildinfo.pri. That would mean that I've to define it somewhere.

[memsharded]

Oh, I see what you mean. Yes, that would only work if you are running from a Conan environment or if you have the variable defined in your environment.

Overall, the main issue is that it is irritating that qt really requires an absolute path that way to be usable.

We will take this feedback into our roadmap, will probably define for Conan 2.0 a way to be able to configure packages post-install in a clean way. I will bring it as a topic for discussion into ConanDays.

[cajus]

Ok. Thanks for the help @memsharded . I guess that question can be closed then ;-)