Pkg.jl: Pkg.add on git repository with a submodule raises GitError

[KristofferC]

@non-Jedi commented on Fri Aug 31 2018

Just upgraded to 1.0 and tried to checkout some of my projects. One has a git submodule in it as a convenience since it's being developed against a spec contained in another git repo, but I am unable to add that one using Pkg:

julia> using Pkg

julia> Pkg.add(PackageSpec(url="https://github.com/non-Jedi/MatrixClientProtocol.jl", rev="master"))
  Updating registry at `~/.julia/registries/General`
  Updating git-repo `https://github.com/JuliaRegistries/General.git`
  Updating git-repo `https://github.com/non-Jedi/MatrixClientProtocol.jl`
ERROR: GitError(Code:ERROR, Class:Submodule, cannot get submodules without a working tree)
Stacktrace:
 [1] macro expansion at /builddir/julia/usr/share/julia/stdlib/v1.0/LibGit2/src/error.jl:101 [inlined]
 [2] #checkout_tree#46(::LibGit2.CheckoutOptions, ::Function, ::LibGit2.GitRepo, ::LibGit2.GitTree) at /builddir/julia/usr/share/julia/stdlib/v1.0/LibGit2/src/repository.jl:358
 [3] #add_or_develop at ./none:0 [inlined]
 [4] #add_or_develop#12 at /builddir/julia/usr/share/julia/stdlib/v1.0/Pkg/src/API.jl:29 [inlined]
 [5] #add_or_develop at ./none:0 [inlined]
 [6] #add_or_develop#10(::Base.Iterators.Pairs{Symbol,Symbol,Tuple{Symbol},NamedTuple{(:mode,),Tuple{Symbol}}}, ::Function, ::Pkg.Types.PackageSpec) at /builddir/julia/usr/share/julia/stdlib/v1.0/Pkg/src/API.jl:27
 [7] #add_or_develop at ./none:0 [inlined]
 [8] #add#18 at /builddir/julia/usr/share/julia/stdlib/v1.0/Pkg/src/API.jl:69 [inlined]
 [9] add(::Pkg.Types.PackageSpec) at /builddir/julia/usr/share/julia/stdlib/v1.0/Pkg/src/API.jl:69
 [10] top-level scope at none:0

To be clear, I'm aware that I should have been using Pkg.develop for what I was trying to do here, but I still think that this shouldn't be raising this error.

[mmolignano]

@KristofferC Have you found any workarounds for this? I've created a project with a dependency on a repo with a submodule. I'm not sure the proper workflow here, but the assumption is that others will use this without using Pkg.develop and I'm not sure how to get around this error.

I was planning on setting up a build script to properly initialize the submodule and build required dependencies within the submodule (that are not part of julia) but I'm not sure the best way to do this.

[marius311]

I'm curious as well. I'd like keep the docs for a given package in a submodule so as not to bloat the main repository, but I get this error when I try to Pkg.add the package (only Pkg.develop works, and clones the repo without the submodules, which is indeed the desired behavior in my case).

[vtjeng]

To clarify: if I have a repository with a submodule, is it (currently) going to be unusable by others who are installing it via Pkg.add?

[korsbo]

It there a way to ignore submodules? In my case, I tend to have scientific projects as an application and the article src as a submodule to that project. Within Julia, I thus do not have any need for the submodule and would like to simply ignore it.

[StefanKarpinski]

We're not going to have multiple possible behaviors for this, so it needs to be decided whether to automatically check out submodules as though they are just subdirectories or ignore them. It's unclear to me at least, based on what people are saying here, which is the better approach.

[korsbo]

If only one behaviour is to be supported I guess that it would make more sense to check out the submodules. It must be easier for devs to stop using submodules for stuff that is not needed for the package itself than to stop using submodules for things that actually matter.

Also, even if the submodule is not needed for the operation of a package it might often not be harmful to check them out anyway (unless it's from a private repo).

[StefanKarpinski]

There's also the consideration that it's impossible to compute (or verify) the tree hash of a tree with submodules that it is not in a full git repository. Even if we had the submodule content in the right place (as a subdirectory), that's not enough information since submodules are represented by git as a commit entry in the tree object. To verify the commit value of a submodule, you'd have and hash all of the history of the submodule. This is why Pkg3 moved away from identifying versions by commit to identifying them by tree hash only. By allowing submodules in published versions of packages, we'd be reintroducing the whole "we need to clone an entire repo just to get any version of it" problem. I'm afraid that's just not viable: published package versions must have independently verifiable tree hashes that only depend on content, not history. That rules out submodules.

[StefanKarpinski]

The bad news is that the solution here is going to have to be developer-side: when publishing a version of a package that uses submodules, some tool will have to create the actual tree that you want to tag as a version, without any submodules. The good news is that tool can do whatever you want: ignore the submodules or create a tree that includes their content. But in any case, not an issue for Pkg itself to deal with. We can support dev mode for repositories with submodules, but that's it. You cannot add a package with submodules since the tree hash cannot be computed.

[non-Jedi]

The bad news is that the solution here is going to have to be developer-side: when publishing a version of a package that uses submodules, some tool will have to create the actual tree that you want to tag as a version, without any submodules.

Is it possible to register a package that isn't a git repository? There's nothing documented to that effect, and as far as I can tell all the packages in the General registry are git repos (but then the structure of Registries isn't documented anywhere in the first place). Or are you saying that to make a release one would have to create a new branch with a commit without the submodules?

I've been wanting to play around with darcs but haven't so far mostly just in case I wanted to register the resulting package at some point...

[StefanKarpinski]

Is it possible to register a package that isn't a git repository?

Not currently, although in the future it will be. The fallback delivery mechanism for packages currently is to git clone a repo and checkout the right tree. Once the Pkg protocol is the primary delivery mechanism, it doesn't matter where the code comes from, it will be decoupled from git.

Or are you saying that to make a release one would have to create a new branch with a commit without the submodules?

Yes, there would have to be somewhere a tagged source tree that is what is installed without any submodules—because submodules cannot be hashed without full git history of the submodules.

[r-barnes]

Today I'm looking at building a simple Julia wrapper around a C++ library. With recursive submodules it's possible to build this very quickly: include the C++ library as a submodule and then construct the necessary Julia files. If the C++ library is later updated, as a developer I now have a clear path for migrating those changes into the Julia library and tracking what's where.

Without the recursive submodules, I'm not even sure where to begin. Copying the entire remote code base is one possibility, but now that code is out of sync with upstream.

[korsbo]

@r-barnes I'm not an expert but I'm guessing that you would be able to use git subtree for this.

[StefanKarpinski]

I explained why supporting git submodules in Pkg.add is _technically_ infeasible: it requires a git clone of every submodules but a package version to install is a tree of files and does not include git history of any kind (we've already done that and it was a disaster). Was there some part of that explanation that was unclear?

You can, however, develop the project however you want using submodules, but then at release time, take a snapshot of the entire work tree, including the content of all submodules, and make that the release. I should also point out that your C++ library use case is exactly what artifacts are for:

https://julialang.org/blog/2019/11/artifacts/

[korsbo]

I know and I thank you for the time you took to explain this! I was, however, suggesting subtrees and not submodules. I have never worked with subtrees myself, so I may be off. But to my understanding, subtrees ensure that everything resides in the main .git folder. For all intents and purposes, it's a normal git project but it still allows you to pull changes from your dependency.

PS. My grammarly plugin is warning me that this text has an angry tone. I don't see that myself but in case grammary is right and I'm wrong I thought I'd be explicit about this not being my intention at all.

[StefanKarpinski]

Agree that git subtree seems like a good solution, I was responding to https://github.com/JuliaLang/Pkg.jl/issues/708#issuecomment-582129565.