Pkg.jl: Can't have a registered package depend on an unregistered repository

[timholy]

It seems impossible currently to safely distribute a registered package with a dependence on an unregistered repository. #492 is a "solution" issue, I thought it might be useful to have one that carefully documents "the problem" so that we can link to a concise description from other packages that are currently thinking about this (motivated by https://github.com/JuliaImages/ImageDistances.jl/pull/4).

Here's an annotated session in which, starting from a fresh package repository, I try to create a new package that depends on an unregistered repository. Here's an English description of the steps I go through:

1. Launch julia so that it uses a blank environment, and install ColorTypes just to populate various directories
2. generate a new package Foo. (Alternatively this could be an existing package to which we want to add something new.)
3. Make Foo depend on https://github.com/robertfeldt/EarthMoversDistance.jl, which is not registered
4. Try to use Foo in the general environment. This is the part that fails.
5. Show that it works OK inside Foo's environment.

If Foo is a "workhorse package" that people use in their primary environment, this makes it impossible to introduce the dependency without telling all the users to directly add or dev EarthMoversDistance in their primary environment. That's an extra step that requires a trip to the README, something people don't generally have to do for packages that depend only on registered packages.

tim@diva:/tmp$ JULIA_DEPOT_PATH=/tmp/pkgs julia -q
(v1.0) pkg> dev ColorTypes
   Cloning default registries into /tmp/pkgs/registries
   Cloning registry General from "https://github.com/JuliaRegistries/General.git"
  Updating registry at `/tmp/pkgs/registries/General`
  Updating git-repo `https://github.com/JuliaRegistries/General.git`
   Cloning git-repo `https://github.com/JuliaGraphics/ColorTypes.jl.git`
  Updating git-repo `https://github.com/JuliaGraphics/ColorTypes.jl.git`
 Resolving package versions...
 Installed FixedPointNumbers ─ v0.5.3
  Updating `/tmp/pkgs/environments/v1.0/Project.toml`
  [3da002f7] + ColorTypes v0.7.5+ [`/tmp/pkgs/dev/ColorTypes`]
  Updating `/tmp/pkgs/environments/v1.0/Manifest.toml`
  [3da002f7] + ColorTypes v0.7.5+ [`/tmp/pkgs/dev/ColorTypes`]
  [53c48c17] + FixedPointNumbers v0.5.3
  [2a0f44e3] + Base64 
  [ade2ca70] + Dates 
  [8ba89e20] + Distributed 
  [b77e0a4c] + InteractiveUtils 
  [76f85450] + LibGit2 
  [8f399da3] + Libdl 
  [37e2e46d] + LinearAlgebra 
  [56ddb016] + Logging 
  [d6f4376e] + Markdown 
  [44cfe95a] + Pkg 
  [de0858da] + Printf 
  [3fa0cd96] + REPL 
  [9a3f8284] + Random 
  [ea8e919c] + SHA 
  [9e88b42a] + Serialization 
  [6462fe0b] + Sockets 
  [8dfed614] + Test 
  [cf7118a7] + UUIDs 
  [4ec0a83e] + Unicode 

shell> cd pkgs/dev
/tmp/pkgs/dev

(v1.0) pkg> generate Foo
Generating project Foo:
    Foo/Project.toml
    Foo/src/Foo.jl

shell> cd Foo
/tmp/pkgs/dev/Foo

(v1.0) pkg> activate .

(Foo) pkg> add https://github.com/robertfeldt/EarthMoversDistance.jl.git
   Cloning git-repo `https://github.com/robertfeldt/EarthMoversDistance.jl.git`
  Updating git-repo `https://github.com/robertfeldt/EarthMoversDistance.jl.git`
[ Info: Assigning UUID e1737d0f-dfce-5227-9318-d7961fac91f7 to EarthMoversDistance
 Resolving package versions...
  Updating `/tmp/pkgs/dev/Foo/Project.toml`
  [e1737d0f] + EarthMoversDistance v0.0.0 #master (https://github.com/robertfeldt/EarthMoversDistance.jl.git)
  Updating `/tmp/pkgs/dev/Foo/Manifest.toml`
  [e1737d0f] + EarthMoversDistance v0.0.0 #master (https://github.com/robertfeldt/EarthMoversDistance.jl.git)
  [2a0f44e3] + Base64 
  [8ba89e20] + Distributed 
  [b77e0a4c] + InteractiveUtils 
  [8f399da3] + Libdl 
  [37e2e46d] + LinearAlgebra 
  [56ddb016] + Logging 
  [d6f4376e] + Markdown 
  [9a3f8284] + Random 
  [9e88b42a] + Serialization 
  [6462fe0b] + Sockets 
  [8dfed614] + Test 
  Building EarthMoversDistance → `/tmp/pkgs/packages/EarthMoversDistance/LbTWZ/deps/build.log`

(Foo) pkg> activate

shell> pwd
/tmp/pkgs/dev/Foo

shell> cd ../..
/tmp/pkgs

(v1.0) pkg> resolve
 Resolving package versions...
  Updating `/tmp/pkgs/environments/v1.0/Project.toml`
 [no changes]
  Updating `/tmp/pkgs/environments/v1.0/Manifest.toml`
 [no changes]

(v1.0) pkg> dev /tmp/pkgs/dev/Foo
 Resolving package versions...
ERROR: Unsatisfiable requirements detected for package EarthMoversDistance [e1737d0f]:
 EarthMoversDistance [e1737d0f] log:
 ├─EarthMoversDistance [e1737d0f] has no known versions!
 └─restricted to versions * by Foo [d11d1dc6] — no versions left
   └─Foo [d11d1dc6] log:
     ├─possible versions are: 0.1.0 or uninstalled
     └─Foo [d11d1dc6] is fixed to version 0.1.0

shell> cd dev/Foo
/tmp/pkgs/dev/Foo

(v1.0) pkg> activate .

julia> using Foo
[ Info: Precompiling Foo [d11d1dc6-ccaf-11e8-00ae-ebbedd466ca6]

julia> Foo.EarthMoversDistance.emd([0.2, 0.7, 0.1], [0.5, 0.5], rand(3, 2))
0.4461155929675431

Here is the state of various files at the end of this process:

tim@diva:/tmp/pkgs$ cat environments/v1.0/Project.toml
[deps]
ColorTypes = "3da002f7-5984-5a60-b8a6-cbb66c0b333f"
tim@diva:/tmp/pkgs$ cat environments/v1.0/Manifest.toml 
[[Base64]]
uuid = "2a0f44e3-6c83-55bd-87e4-b1978d98bd5f"

[[ColorTypes]]
deps = ["FixedPointNumbers", "Random", "Test"]
path = "/tmp/pkgs/dev/ColorTypes"
uuid = "3da002f7-5984-5a60-b8a6-cbb66c0b333f"
version = "0.7.5+"

[[Dates]]
deps = ["Printf"]
uuid = "ade2ca70-3891-5945-98fb-dc099432e06a"

[[Distributed]]
deps = ["LinearAlgebra", "Random", "Serialization", "Sockets"]
uuid = "8ba89e20-285c-5b6f-9357-94700520ee1b"

[[FixedPointNumbers]]
deps = ["Pkg", "Test"]
git-tree-sha1 = "b8045033701c3b10bf2324d7203404be7aef88ba"
uuid = "53c48c17-4a7d-5ca2-90c5-79b7896eea93"
version = "0.5.3"

[[InteractiveUtils]]
deps = ["LinearAlgebra", "Markdown"]
uuid = "b77e0a4c-d291-57a0-90e8-8db25a27a240"

[[LibGit2]]
uuid = "76f85450-5226-5b5a-8eaa-529ad045b433"

[[Libdl]]
uuid = "8f399da3-3557-5675-b5ff-fb832c97cbdb"

[[LinearAlgebra]]
deps = ["Libdl"]
uuid = "37e2e46d-f89d-539d-b4ee-838fcccc9c8e"

[[Logging]]
uuid = "56ddb016-857b-54e1-b83d-db4d58db5568"

[[Markdown]]
deps = ["Base64"]
uuid = "d6f4376e-aef5-505a-96c1-9c027394607a"

[[Pkg]]
deps = ["Dates", "LibGit2", "Markdown", "Printf", "REPL", "Random", "SHA", "UUIDs"]
uuid = "44cfe95a-1eb2-52ea-b672-e2afdf69b78f"

[[Printf]]
deps = ["Unicode"]
uuid = "de0858da-6303-5e67-8744-51eddeeeb8d7"

[[REPL]]
deps = ["InteractiveUtils", "Markdown", "Sockets"]
uuid = "3fa0cd96-eef1-5676-8a61-b3b8758bbffb"

[[Random]]
deps = ["Serialization"]
uuid = "9a3f8284-a2c9-5f02-9a11-845980a1fd5c"

[[SHA]]
uuid = "ea8e919c-243c-51af-8825-aaa63cd721ce"

[[Serialization]]
uuid = "9e88b42a-f829-5b0c-bbe9-9e923198166b"

[[Sockets]]
uuid = "6462fe0b-24de-5631-8697-dd941f90decc"

[[Test]]
deps = ["Distributed", "InteractiveUtils", "Logging", "Random"]
uuid = "8dfed614-e22c-5e08-85e1-65c5234f0b40"

[[UUIDs]]
deps = ["Random"]
uuid = "cf7118a7-6976-5b1a-9a39-7adc72f591a4"

[[Unicode]]
uuid = "4ec0a83e-493e-50e2-b9ac-8f72acf5a8f5"
tim@diva:/tmp/pkgs$ cat dev/Foo/Project.toml 
name = "Foo"
uuid = "d11d1dc6-ccaf-11e8-00ae-ebbedd466ca6"
authors = ["Tim Holy <[email protected]>"]
version = "0.1.0"

[deps]
EarthMoversDistance = "e1737d0f-dfce-5227-9318-d7961fac91f7"
tim@diva:/tmp/pkgs$ cat dev/Foo/Manifest.toml 
[[Base64]]
uuid = "2a0f44e3-6c83-55bd-87e4-b1978d98bd5f"

[[Distributed]]
deps = ["LinearAlgebra", "Random", "Serialization", "Sockets"]
uuid = "8ba89e20-285c-5b6f-9357-94700520ee1b"

[[EarthMoversDistance]]
deps = ["Libdl", "Test"]
git-tree-sha1 = "8278e0f422eb8a884e385c74fa015573eb36df8a"
repo-rev = "master"
repo-url = "https://github.com/robertfeldt/EarthMoversDistance.jl.git"
uuid = "e1737d0f-dfce-5227-9318-d7961fac91f7"
version = "0.0.0"

[[InteractiveUtils]]
deps = ["LinearAlgebra", "Markdown"]
uuid = "b77e0a4c-d291-57a0-90e8-8db25a27a240"

[[Libdl]]
uuid = "8f399da3-3557-5675-b5ff-fb832c97cbdb"

[[LinearAlgebra]]
deps = ["Libdl"]
uuid = "37e2e46d-f89d-539d-b4ee-838fcccc9c8e"

[[Logging]]
uuid = "56ddb016-857b-54e1-b83d-db4d58db5568"

[[Markdown]]
deps = ["Base64"]
uuid = "d6f4376e-aef5-505a-96c1-9c027394607a"

[[Random]]
deps = ["Serialization"]
uuid = "9a3f8284-a2c9-5f02-9a11-845980a1fd5c"

[[Serialization]]
uuid = "9e88b42a-f829-5b0c-bbe9-9e923198166b"

[[Sockets]]
uuid = "6462fe0b-24de-5631-8697-dd941f90decc"

[[Test]]
deps = ["Distributed", "InteractiveUtils", "Logging", "Random"]
uuid = "8dfed614-e22c-5e08-85e1-65c5234f0b40"
tim@diva:/tmp/pkgs$ 

[KristofferC]

Just to perhaps point out the obvious, this is not a regression vs the old package manager but a feature request, right?

To give some background to the complications here, it is worth asking the question "why do we have a registry at all"? Can't we just add all packages by URL and just discover what dependencies they have, rinse and repeat recursively? The reason for having a registry is to be able to do version resolution without downloading the world. If we had direct access to the whole world of packages we could just read their dependencies and compatibility for each tagged version and resolve. But people tend to want to use the package manager without downloading every single Julia package, and therefore we have a registry containing the needed information.

So there are multiple questions that need to be answered here:

• How should a dependency on an unregistered package be declared (just an url in the project file?)
• How do you declare what version compatibility you have with that unregistered package when it might not even have any versions?
• When doing version resolution, how can we ensure a consistent result when we have no dependency information for the unregistered package?

and so on.

Note that instantiating a Manifest doesn't even touch the resolver so there it is a different story.

[tpapp]

Sorry for being obtuse, but I have just reread #492 and I still don't know what the solution is (apparently having a Manifest.toml is not enough for your case).

@KristofferC: you raise valid points. Yet for just working with WIP packages, eg the following would suffice for my usual workflow:

1. the user declares the URL for a git repo,
2. the master of which is the only version that is considered,
3. if dependencies can't be satisfied with the Project.toml of these packages in master, then the version resolver is allowed to give up.

If one wants version management and fine-grained dependency control with versions, they should register.

[timholy]

Just to perhaps point out the obvious, this is not a regression vs the old package manager but a feature request, right?

Right. The problem is that Pkg currently offers just enough features that move in this direction that the current state feels incomplete: I can create a project and declare its (internal) dependency on some random repo, but then I can't easily incorporate that project into my daily workflow or start pushing things up to GitHub. I'm not sure whether it's a documentation issue, a feature issue, or what, but I do think we need to do something soon to fix the niggling issues that affect people who develop a lot of packages. (#770 is basically another report along the same lines.) @KristofferC, I know you're already moving mountains and that you're already pushing the limits of what's humanly possible.

To broaden the discussion a bit, we've actually created our own private registry for my lab's private code. Mostly, this appears to be a success. I say "appears" because most users are still stuck on 0.6, as the upgrades of our large internal code base have been tricky and taken a long time. As a consequence there hasn't been enough real-world testing to know how it will work out. I'll file a separate issue about my experiences so far.

With regards to

How do you declare what version compatibility you have with that unregistered package when it might not even have any versions

If we were to do something like #492 then I'd say "let's not have version control," since to me versions are something that implies a registry. So if you use this feature you just use the latest release. The alternative is to make registry-creation dirt simple, which it currently isn't.

I don't think we necessarily need both: either "easy registries" or "support URLs" seem like it could work, with the registries of course being the more powerful approach. Is there a commitment to that route, or are people still unsure?

[KristofferC]

The alternative is to make registry-creation dirt simple, which is currently isn't.

True, slowly poking away on it though, https://github.com/JuliaLang/PkgDev.jl/pull/144.

I don't think we can ever have that installing a versioned package can lead to unregistered packages being installed, but installing a package from a URL could potentially recursively pull down other URL dependencies and consider the version of all those as fixed.

[Gregstrq]

I have noticed that issue is still open after two years. As a consequence, I would like to ask: what is the status of this issue?

If a registered package were to depend on an unregistered one, would it possible to make it happen by requiring two things:

1. replace uuid by direct repo url (since url and uuid have different format, the thing that parses Project.toml can distinguish both.)
2. in [compat] section, replace version range by a range of commits (or, may be, allow to specify only a single commit).

[KristofferC]

See https://github.com/JuliaLang/Pkg.jl/issues/810#issuecomment-428676759. The issues there still exist.

[Gregstrq]

It looks like the real problem is paragraph 3: how to check the dependencies of the unregistered package.

Theoretically, it should be possible just to parse Project.toml file of the unregistered package on the go. There is a good chance, that dependencies of the unregistered package would be registered. As far as I understand, if there is a deep chain of unregistered dependencies, one should really create a local registry instead. However, this might be very slow.

May it possible to limit the depth of the chain of unregistered dependencies (for example, the unregistered dependency should have only registered dependencies; or it may have unregistered dependencies, but they can have only registered dependencies)? Then, the presence of Project.toml in unregistered dependency, as well as the depth of the chain of unregistered dependencies can be checked when the package is registered.

[Gregstrq]

In this scheme, the paragraphs 1 and 2 of my post cover the paragraphs 1 and 2 of the original issues.

[StefanKarpinski]

Sounds complicated any annoying to implement and test.

[Gregstrq]

Sounds complicated any annoying to implement and test.

Did you mean here all my points, or there is some specific part, which is particularly hard to implement and test? (Just trying to understand if anything can be salvaged from my idea.)

[StefanKarpinski]

I mean the whole business of trying to install things and then backtracking and having arbitrary cutoffs for how deep you'll go or how much backtracking is allowed. Seems like a bad situation all around. Which would be fine if someone else were going to try out implementing it and maintain it later, but this proposal seems like a request for Pkg devs to develop and maintain it... and I'm just very disinclined from doing any of that. Moreover, every time I try to think through how this could reasonably work, every time I end up at the conclusion that this would add an absurd amount of complexity and it would just be much more reasonable for people to register their dependencies.