Julia: Pkg.update() fails on FreeBSD due to libgit2 issues

[ararslan]

   _       _ _(_)_     |  A fresh approach to technical computing
  (_)     | (_) (_)    |  Documentation: https://docs.julialang.org
   _ _   _| |_  __ _   |  Type "?help" for help.
  | | | | | | |/ _` |  |
  | | |_| | | | (_| |  |  Version 0.7.0-DEV.887 (2017-07-10 18:22 UTC)
 _/ |\__'_|_|_|\__'_|  |  Commit 86395faee1* (0 days old master)
|__/                   |  x86_64-unknown-freebsd11.0

julia> Pkg.update()
INFO: Updating METADATA...
ERROR: METADATA cannot be updated. Resolve problems manually in /home/alex/.julia/v0.7/METADATA.
GitError(Code:ERROR, Class:SSL, there is no TLS stream available)
macro expansion at ./libgit2/error.jl:99 [inlined]
#fetch#51(::Base.LibGit2.FetchOptions, ::String, ::Function, ::Base.LibGit2.GitRemote, ::Array{AbstractString,1}) at ./libgit2/remote.jl:291
(::Base.LibGit2.#kw##fetch)(::Array{Any,1}, ::Base.LibGit2.#fetch, ::Base.LibGit2.GitRemote, ::Array{AbstractString,1}) at ./<missing>:0
#fetch#95(::String, ::String, ::Array{AbstractString,1}, ::Nullable{Base.LibGit2.AbstractCredentials}, ::Function, ::Base.LibGit2.GitRepo) at ./libgit2/libgit2.jl:277
(::Base.Pkg.Entry.##33#39)(::Base.LibGit2.GitRepo) at ./pkg/entry.jl:371
with at ./libgit2/types.jl:775 [inlined]
with at ./libgit2/types.jl:781 [inlined]
update(::String, ::Set{String}) at ./pkg/entry.jl:354
#4 at ./pkg/dir.jl:36 [inlined]
cd(::Base.Pkg.Dir.##4#7{Array{Any,1},Base.Pkg.Entry.#update,Tuple{String,Set{String}}}, ::String) at ./file.jl:70
#cd#1(::Array{Any,1}, ::Function, ::Function, ::String, ::Vararg{Any,N} where N) at ./pkg/dir.jl:36
update() at ./pkg/pkg.jl:228
eval(::Module, ::Expr) at ./repl/REPL.jl:3
eval_user_input(::Any, ::Base.REPL.REPLBackend) at ./repl/REPL.jl:67
macro expansion at ./repl/REPL.jl:98 [inlined]
(::Base.REPL.##1#2{Base.REPL.REPLBackend})() at ./event.jl:73
Stacktrace:
 [1] (::Base.Pkg.Entry.##33#39)(::Base.LibGit2.GitRepo) at ./pkg/entry.jl:378
 [2] with at ./libgit2/types.jl:775 [inlined]
 [3] with at ./libgit2/types.jl:781 [inlined]
 [4] update(::String, ::Set{String}) at ./pkg/entry.jl:354
 [5] #4 at ./pkg/dir.jl:36 [inlined]
 [6] cd(::Base.Pkg.Dir.##4#7{Array{Any,1},Base.Pkg.Entry.#update,Tuple{String,Set{String}}}, ::String) at ./file.jl:70
 [7] #cd#1(::Array{Any,1}, ::Function, ::Function, ::String, ::Vararg{Any,N} where N) at ./pkg/dir.jl:36
 [8] update() at ./pkg/pkg.jl:228

May be related to the recent libgit2 upgrade. If so, we may need to check whether there are any libgit2 patches in the FreeBSD Ports system that we'll need to pull in. EDIT: The libgit2 port hasn't been updated for the newest version yet.

[iblis17]

I sent a patch for upgrading devel/libgit2
https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=220620

[iblis17]

I pull Julia master and build it with system libgit2 v0.26.0.
The Pkg.update works fine.

and I confirmed that the private libgit2 got the same error as @ararslan

[iblis17]

diff --git a/deps/libgit2.mk b/deps/libgit2.mk
index d83d0864c3..90ae2e0df1 100644
--- a/deps/libgit2.mk
+++ b/deps/libgit2.mk
@@ -40,7 +40,7 @@ ifeq ($(OS),Linux)
 LIBGIT2_OPTS += -DUSE_HTTPS=ON -DTLS_BACKEND="mbedTLS" -DCMAKE_INSTALL_RPATH="\$$ORIGIN"
 endif
 ifeq ($(OS),FreeBSD)
-LIBGIT2_OPTS += -DCMAKE_INSTALL_RPATH="\$$ORIGIN"
+LIBGIT2_OPTS += -DTLS_BACKEND="mbedTLS" -DCMAKE_INSTALL_RPATH="\$$ORIGIN"
 endif

 # We need to bundle ca certs on linux now that we're using libgit2 with ssl
@@ -61,6 +61,11 @@ CERTFILE=$(OPENSSL_DIR)/certs/ca.pem
 endif
 endif # Linux

+ifeq ($(OS),FreeBSD)
+OPENSSL_DIR=$(shell openssl version -d | cut -d '"' -f 2)
+CERTFILE=$(OPENSSL_DIR)/cert.pem
+endif # FreeBSD
+
 LIBGIT2_SRC_PATH := $(SRCDIR)/srccache/$(LIBGIT2_SRC_DIR)

 $(LIBGIT2_SRC_PATH)/libgit2-ssh.patch-applied: $(LIBGIT2_SRC_PATH)/source-extracted

[tkelman]

That looks PR-worthy to me. How did it work at all before without setting a TLS backend?

[iblis17]

Seems this commit delete the src/openssl_stream.c which serve as SSL backend on FreeBSD https://github.com/JuliaLang/julia/commit/8d47a314537779c8fb86642c54925613628a91b0#diff-7b94fb465bb67b40e70933b8f97f7d65R903

[tkelman]

No, it was just moved to src/streams/openssl.c

[iblis17]

(hmm... still not found the reason why it worked before :/

[omus]

@iblis17 I think your patch needs to include USE_HTTPS

 ifeq ($(OS),FreeBSD)
-LIBGIT2_OPTS += -DCMAKE_INSTALL_RPATH="\$$ORIGIN"
+LIBGIT2_OPTS += -DUSE_HTTPS=ON -DTLS_BACKEND="mbedTLS" -DCMAKE_INSTALL_RPATH="\$$ORIGIN"
 endif

See https://github.com/JuliaLang/julia/commit/8d47a314537779c8fb86642c54925613628a91b0#diff-a01ad0cac7aa5a648c5d00ec39bd9afcR40

[omus]

Definitely my bad that FreeBSD got left out during the update to LibGit2 v0.26.0. Probably these Linux and FreeBSD options should be combined: https://github.com/JuliaLang/julia/blob/8d47a314537779c8fb86642c54925613628a91b0/deps/libgit2.mk#L39-L44