Pkg.jl: Private registries: experiences so far

[timholy]

We've created our own registry for lab-private code. Compared to 0.6, this already offers some advantages in that once you install it, you get by-name access to all the packages it contains. That's lovely.

Below I describe a few issues we've noticed in our limited usage so far.

Registry updates

If you're modifying the registry itself and on a branch, you often get either warnings or errors. For example, if I'm on the teh/https branch of HolyLabRegistry (the name of our private registry), and this branch is local to my machine, I can get this: (here TileTrees is one of our private repositories and BoxTrees is one of its private dependencies, both of which are registered in HolyLabRegistry):

(v1.0) pkg> add TileTrees
  Updating registry at `/tmp/pkgs2/registries/General`
  Updating git-repo `https://github.com/JuliaRegistries/General.git`
  Updating registry at `/tmp/pkgs2/registries/HolyLabRegistry`
  Updating git-repo `https://github.com/HolyLab/HolyLabRegistry.git`
Username for 'https://github.com': timholy
Password for 'https://[email protected]': 
┌ Warning: Some registries failed to update:
│     — /tmp/pkgs2/registries/HolyLabRegistry — branch origin/teh/https not found
└ @ Pkg.API ~/src/julia-1.0/usr/share/julia/stdlib/v1.0/Pkg/src/API.jl:157
 Resolving package versions...
 Installed Conda ───────────────────── v1.0.2
 Installed ImageDistances ──────────── v0.1.1
 Installed OrderedCollections ──────── v1.0.2
... # lots of packages
 Installed StaticArrays ────────────── v0.8.3
[ Info: Installed TileTrees ───────────────── v0.0.0
   Cloning [9cc36475-e10c-57f8-b33d-19dcb253ebc3] BoxTrees from https://github.com/HolyLab/BoxTrees.jl.git
Username for 'https://github.com': timholy
Password for 'https://[email protected]': 
[ Info: Installed BoxTrees ────────────────── v0.0.0
  Updating `/tmp/pkgs2/environments/v1.0/Project.toml`
  [cd6f60fb] + TileTrees v0.0.0
  Updating `/tmp/pkgs2/environments/v1.0/Manifest.toml`
  [621f4979] + AbstractFFTs v0.3.2
  [13072b0f] + AxisAlgorithms v0.3.0
...  # lots
  [1a1011a3] + SharedArrays 
  Building Conda ───────────→ `/tmp/pkgs2/packages/Conda/hsaaN/deps/build.log`
  Building FFTW ────────────→ `/tmp/pkgs2/packages/FFTW/p7sLQ/deps/build.log`
  Building SpecialFunctions → `/tmp/pkgs2/packages/SpecialFunctions/sXbz6/deps/build.log`
  Building TileTrees ───────→ `/tmp/pkgs2/packages/TileTrees/XIM0y/deps/build.log`
┌ Error: Error building `TileTrees`: 
│ ┌ Warning: Some registries failed to update:
│ │     — /tmp/pkgs2/registries/HolyLabRegistry — failed to fetch from repo
│ └ @ Pkg.API ~/src/julia-1.0/usr/share/julia/stdlib/v1.0/Pkg/src/API.jl:157
│ ERROR: LoadError: failed to clone from [email protected]:HolyLab/BoxTrees.jl.git, error: GitError(Code:EUSER, Class:Callback, Aborting, user cancelled credential request.)
│ Stacktrace:
│  [1] pkgerror(::String) at /home/tim/src/julia-1.0/usr/share/julia/stdlib/v1.0/Pkg/src/Types.jl:120
│  [2] #clone#2(::Nothing, ::Base.Iterators.Pairs{Symbol,Any,Tuple{Symbol,Symbol},NamedTuple{(:isbare, :credentials),Tuple{Bool,LibGit2.CachedCredentials}}}, ::Function, ::String, ::String) at /home/tim/src/julia-1.0/usr/share/julia/stdlib/v1.0/Pkg/src/GitTools.jl:107
│  [3] (::getfield(Pkg.GitTools, Symbol("#kw##clone")))(::NamedTuple{(:isbare, :credentials),Tuple{Bool,LibGit2.CachedCredentials}}, ::typeof(Pkg.GitTools.clone), ::String, ::String) at ./none:0
│  [4] #handle_repos_add!#32(::Bool, ::Nothing, ::Function, ::Pkg.Types.Context, ::Array{Pkg.Types.PackageSpec,1}) at /home/tim/src/julia-1.0/usr/share/julia/stdlib/v1.0/Pkg/src/Types.jl:627
│  [5] #handle_repos_add! at ./none:0 [inlined]
│  [6] #add_or_develop#13(::Symbol, ::Bool, ::Base.Iterators.Pairs{Union{},Union{},Tuple{},NamedTuple{(),Tuple{}}}, ::Function, ::Pkg.Types.Context, ::Array{Pkg.Types.PackageSpec,1}) at /home/tim/src/julia-1.0/usr/share/julia/stdlib/v1.0/Pkg/src/API.jl:54
│  [7] #add_or_develop#12 at ./none:0 [inlined]
│  [8] #add_or_develop at ./none:0 [inlined]
│  [9] #add_or_develop#10 at /home/tim/src/julia-1.0/usr/share/julia/stdlib/v1.0/Pkg/src/API.jl:27 [inlined]
│  [10] #add_or_develop at ./none:0 [inlined]
│  [11] #add#18 at /home/tim/src/julia-1.0/usr/share/julia/stdlib/v1.0/Pkg/src/API.jl:69 [inlined]
│  [12] add(::Pkg.Types.PackageSpec) at /home/tim/src/julia-1.0/usr/share/julia/stdlib/v1.0/Pkg/src/API.jl:69
│  [13] top-level scope at none:0
│  [14] include at ./boot.jl:317 [inlined]
│  [15] include_relative(::Module, ::String) at ./loading.jl:1041
│  [16] include(::Module, ::String) at ./sysimg.jl:29
│  [17] include(::String) at ./client.jl:388
│  [18] top-level scope at none:0
│ in expression starting at /tmp/pkgs2/packages/TileTrees/XIM0y/deps/build.jl:3
│   Updating registry at `/tmp/pkgs2/registries/General`
│   Updating git-repo `https://github.com/JuliaRegistries/General.git`
    Updating registry at `/tmp/pkgs2/registries/HolyLabRegistry`
│   Updating git-repo `https://github.com/HolyLab/HolyLabRegistry.git`
                                        Cloning git-repo `[email protected]:HolyLab/BoxTrees.jl.git`

└ @ Pkg.Operations ~/src/julia-1.0/usr/share/julia/stdlib/v1.0/Pkg/src/Operations.jl:1069

Authentication

When I add a package that depends on a second package, I have to provide my GitHub password or ssh passphrase frequently. Moreover, we need a .travis.yml file with a script section like this:

 script:
  - if [[ -a .git/shallow ]]; then git fetch --unshallow; fi
  - julia -e 'using Pkg, LibGit2;
              user_regs = joinpath(DEPOT_PATH[1],"registries");
              mkpath(user_regs);
              all_registries = Dict("General" => "https://github.com/JuliaRegistries/General.git",
                            "HolyLabRegistry" => "[email protected]:HolyLab/HolyLabRegistry.git");
              Base.shred!(LibGit2.CachedCredentials()) do creds
                for (reg, url) in all_registries
                  path = joinpath(user_regs, reg);
                  LibGit2.with(Pkg.GitTools.clone(url, path; header = "registry $reg from $(repr(url))", credentials = creds)) do repo end
                end
              end'
- julia -e 'using Pkg; Pkg.build(); Pkg.test'

which does not exactly roll off the tongue.

Version control

Some packages depend on specific branches of registered public projects. For example, https://github.com/mlewe/BlossomV.jl/pull/13 is a dependency for us, but that PR is languishing. That basically breaks testing and merging for dependent projects (including one public one, https://github.com/timholy/CoordinateSplittingPTrees.jl/pull/2). I'd like to have some good way of declaring a specific branch, perhaps in an alternate repo, without breaking the notion of "upstream" (at least, not yet). And, how to handle this at the level of version numbers? If I name my specific branch v0.4.0, what happens when the real v0.4.0 comes out?

[ghost]

Currently one thing that I like about the registry system is that registries are easy enough to clone and manually add to your julia install. You don't have to start julia at all to add a registry, which makes it easy to advertise your custom registry on any projects that rely upon it.

It looks like PR #588 will make this less easy to do as it will add the registry into an inner folder with some random garbage name, similar to the approach used for handling multiple packages.

https://github.com/JuliaLang/Pkg.jl/pull/588

pkg> registry add https://github.com/JuliaRegistries/General
   Cloning registry from "https://github.com/JuliaRegistries/General"
     Added registry `General` to `~/.julia/registries/General/SxLXF`  <---- ewwwww

I wrote a small tool for creating registries and adding projects to them. It's pretty sparse and mostly tailored to my needs; i can take a bit of time to clean it up and update it to the current state of master if anyone would find it useful.

Some packages depend on specific branches of registered public projects. For example, mlewe/BlossomV.jl#13 is a dependency for us, but that PR is languishing.

If there was a dependency on a fork shared by your entire lab, it'd be nifty if you could use your registry to point references to that UUID to your newer fork, given some kind of precedence rule for selecting your registry over General. That way anyone who configured the lab's registry appropriately would automatically get that fix when they add the project. If there was better support for overlapping registries, that could be a thing. I opened an issue about this a while ago. #533

[fredrikekre]

It looks like PR #588 will make this less easy to do

588 is about making adding registries simpler so... Do you really find that manually git-cloning into .julia/registries is simpler than

pkg> registry add https://github.com/JuliaRegistries/General

?

[ghost]

If registry add https://whatever did a simple git clone into the depot, it would be simpler than doing a clone into the depot into a differently named folder nested within the name of the registry. (Not least because you'd always know what registry was being used for your install, rather than having to decode arcane character strings) The nice side effect is that you could add registries out of band manually (maybe for CI for example) without having to run through the julia process.

I totally buy that approach makes sense for packages, where there's lots of cases where you could have multiple versions of the package on a given system, but I have a harder time understanding the appeal of having multiple versions of the same registry in differing folders. Have you used registries that way in your personal work?

EDIT: I noticed that 588 got merged. I'm glad there's meaningful improvements in that PR, but i hope the question about where registries live isn't fully set in stone yet; i feel there's disadvantages to that design. I regret not commenting or opening an issue about that part of the PR when i could have earlier.

[00vareladavid]

I noticed that 588 got merged

? It has not been merged yet

[ghost]

ah my github notifications were a lie! (I think i was confused ;))

[fredrikekre]

The nice side effect is that you could add registries out of band manually (maybe for CI for example) without having to run through the julia process.

IMO

- julia -e 'using Pkg; pkg"registry add  www.my-registry.com'

is simpler than something like

- pushd ~/.julia/registries && git clone www.my-registry.com MyRegistry && popd

especially since, by using Pkg for this, it will be platform independent etc.

I totally buy that approach makes sense for packages, where there's lots of cases where you could have multiple versions of the package

This is actually not what is happening. The reason is that there might be multiple packages with the same name, and we would like to be able to have different registries with the same name too, differentiated by uuid. I think it would be better if you voiced your opinions about #588 over there.

[StefanKarpinski]

we would like to be able to have different registries with the same name too, differentiated by uuid

I do not share that goal. I'm not sure where the idea to do this came from... We do not need to be able to support multiple registries with the same name. Especially because the name of a registry is totally insignificant. Want to have two different registries with the same name? Rename one of them. Literally nothing changes—the system doesn't care at all what the names of registries are.

[KristofferC]

What is there to do here?

[timholy]

Thanks to improvements in authentication and @GunnarFarneback 's LocalRegistry.jl I think we're pretty happy with our registry maintenance now. I haven't tried the branch dependency thing in a while, so I can't comment on that.

[StefanKarpinski]

Seems like this can probably be closed then and more specific issue opened for things that are still rough.