Conan-center-index: [question] How to package libraries that contain git submodules?

[bmanga]

Using a zipped source of a library from github is not an option if it uses submodules as these are not provided in the zip. However by downloading as a zip we also lose the git metadata, so we can't do a "git submodule update" from the recipes' source method.

How do you plan to deal with these cases?

[danimtb]

I think you would have to clode the sources in the recipe source() method the same way you'd do it manually. Something like git clone --recursive <url> should work

[bmanga]

The instructions seem to imply that the sources must come from zipped archives with provided checksums. There is a specific section for github sources and how you should copy the link from the Download ZIP button.

Would it be possible to relax the restriction to be able to reference a git repository at a specific hash commit?

[Croydon]

If the submodules are libraries for which it is reasonable to package them on their own, we did ignore the submodules and provided the libraries as Conan dependencies in the past

[bmanga]

That would be of course ideal, but some larger, popular libraries have a large number of submodule dependencies with very few being already packaged, some of them fairly obscure, and some recursively requiring submodules to build.
As an example, I am trying to package pytorch, but first creating a package for each submodule would take far too long: https://github.com/pytorch/pytorch/tree/master/third_party

[danimtb]

My suggestion would be to use the git url and use a git commit instead of the sha2 checksum in the conandata.yml. Then you can use those values in the recipe to clone the repo with its submodules.

[danimtb]

Coming back to this issue, I think that trying to create recipes of git submodules in case those are C/C++ libraries is the correct way to go.

If any of that is not possible at the moment of the PR, maybe is better to ask the community for help or, if the recipe is already too complex or submodules are ony for internal use (embedded in the library), we might consider to include the recipe and develop it further in the future.

BTW, you can get a zip file from the source code of GitHub repo by its commit, for example https://github.com/pytorch/pytorch/archive/ff608a9.zip. A repo with submodules could point to the URLs of the submodules and recreate the sources' layout in the same way a git clone with submodules would do it

[uilianries]

BTW Conan offers Git helper. If you need to run git submodule command, will be better using the helper instead of self.run("git ...")

[danimtb]

Closing this as aswered. Do not hesitate to open a new issue if you have any other question. Thanks!