Julia: Check validity of arguments to Pkg.init

[simonbyrne]

I don't know if I'm the only one, but sometimes I accidentally call Pkg.init instead of Pkg.generate, which then sets the remote URL of METADATA incorrectly:

julia> Pkg.init("xxx")
INFO: Initializing package repository /Users/simon/.julia/v0.5
INFO: Package directory /Users/simon/.julia/v0.5 is already initialized.

julia> Pkg.update()
INFO: Updating METADATA...
ERROR: METADATA cannot be updated. Resolve problems manually in /Users/simon/.julia/v0.5/METADATA.
GitError(Code:ERROR, Class:Net, Unsupported URL protocol)
 in macro expansion at ./libgit2/error.jl:98 [inlined]
 in #fetch#52(::Base.LibGit2.FetchOptions, ::String, ::Function, ::Base.LibGit2.GitRemote, ::Array{AbstractString,1}) at ./libgit2/remote.jl:70
 in (::Base.LibGit2.#kw##fetch)(::Array{Any,1}, ::Base.LibGit2.#fetch, ::Base.LibGit2.GitRemote, ::Array{AbstractString,1}) at ./<missing>:0
 in #fetch#91(::String, ::String, ::Array{AbstractString,1}, ::Nullable{Base.LibGit2.AbstractPayload}, ::Function, ::Base.LibGit2.GitRepo) at ./libgit2/libgit2.jl:159
 in (::Base.Pkg.Entry.##35#41)(::Base.LibGit2.GitRepo) at ./pkg/entry.jl:371
 in with(::Base.Pkg.Entry.##35#41, ::Base.LibGit2.GitRepo) at ./libgit2/types.jl:660
 in update(::String, ::Set{String}) at ./pkg/entry.jl:354
 in (::Base.Pkg.Dir.##2#3{Array{Any,1},Base.Pkg.Entry.#update,Tuple{String,Set{String}}})() at ./pkg/dir.jl:31
 in cd(::Base.Pkg.Dir.##2#3{Array{Any,1},Base.Pkg.Entry.#update,Tuple{String,Set{String}}}, ::String) at ./file.jl:59
 in #cd#1(::Array{Any,1}, ::Function, ::Function, ::String, ::Vararg{Any,N}) at ./pkg/dir.jl:31
 in update() at ./pkg/pkg.jl:210
 in eval(::Module, ::Any) at ./boot.jl:234
 in eval_user_input(::Any, ::Base.REPL.REPLBackend) at ./REPL.jl:62
 in macro expansion at ./REPL.jl:92 [inlined]
 in (::Base.REPL.##1#2{Base.REPL.REPLBackend})() at ./event.jl:46
 in (::Base.Pkg.Entry.##35#41)(::Base.LibGit2.GitRepo) at ./pkg/entry.jl:378
 in with(::Base.Pkg.Entry.##35#41, ::Base.LibGit2.GitRepo) at ./libgit2/types.jl:660
 in update(::String, ::Set{String}) at ./pkg/entry.jl:354
 in (::Base.Pkg.Dir.##2#3{Array{Any,1},Base.Pkg.Entry.#update,Tuple{String,Set{String}}})() at ./pkg/dir.jl:31
 in cd(::Base.Pkg.Dir.##2#3{Array{Any,1},Base.Pkg.Entry.#update,Tuple{String,Set{String}}}, ::String) at ./file.jl:59
 in #cd#1(::Array{Any,1}, ::Function, ::Function, ::String, ::Vararg{Any,N}) at ./pkg/dir.jl:31
 in update() at ./pkg/pkg.jl:210
 in eval(::Module, ::Any) at ./boot.jl:234
 in macro expansion at ./REPL.jl:92 [inlined]
 in (::Base.REPL.##1#2{Base.REPL.REPLBackend})() at ./event.jl:46

Now, this can easily be fixed by simply calling Pkg.init() but should we also,

a) check that the argument to Pkg.init is valid, and/or
b) provide a more helpful error message to Pkg.update()?

I spent a few minutes poking around libgit2, and I couldn't figure out an obvious way to do (a) (which is why this is an issue and not a PR).

[wildart]

Do you have latest master? Use latest build of libgit2 which supports SSL & SSH. You could check this by running LibGit2.features().

[simonbyrne]

I get:

julia> LibGit2.features()
3-element Array{Base.LibGit2.Consts.GIT_FEATURE,1}:
 FEATURE_THREADS
 FEATURE_HTTPS  
 FEATURE_SSH    

How does that help?

[wildart]

Sorry, got confused. Pkg.init first parameter corresponds to METADATA remote repo location. We definitely need url check in Pkg.init function. As for Pkg.update, its behavior is quite legit. It has nothing to do with libgit2, more of Pkg problem. What can you expect from the url that starts with "xxx"?

As I ecall, there are at least two more places in Base that require url sanity check. We should think about adding url parser to Base.

[simonbyrne]

Should we check that the argument to Pkg.init is (a) a valid URL, and (b) points to a git repository?

[wildart]

Definitely need to check validity of URL, as for repository validation - no luck here. Pkg.init does not do any remote operations, only sets up local repo. So any operation that would later call fetch on METADATA would fail.

We can also try to download and checkout METADATA during Pkg.init call.

[tkelman]

We can also try to download and checkout METADATA during Pkg.init call.

Don't we usually do that, at least in the default 0-arg case?

[simonbyrne]

Only if it hasn't been init-ed already, otherwise we just change the remote url.

[omus]

After working on #20797 I find that having a second call to Pkg.init modify the URL/branch is rather strange and could easily happen accidentally. Someone using a custom METADATA could easily do the following:

Pkg.init("URL", "branch")  # Set the remote and branch
Pkg.init()  # Accidentally set the remote and branch back to the default

Maybe it would be best to introduce a keyword like force which overrides the remote and branch of the METADATA only when force=true. Hypothetical example:

julia> Pkg.init("URL", "branch")
INFO: Initializing package repository ~/.julia/v0.5
INFO: Cloning METADATA from URL

julia> Pkg.init()  # Changes nothing
INFO: Package directory ~/.julia/v0.5 is already initialized. Use `force=true` to modify.

julia> Pkg.init(force=true)
INFO: Updating METADATA remote to be https://github.com/JuliaLang/METADATA.jl

[simonbyrne]

No longer an issue due to new Pkg.