Julia: LibGit2.set_ssl_cert_locations segfaults on a cluster

[andreasnoack]

I got the #15128 error on a cluster when running Pkg.init() and when trying to point set_ssl_cert_locations to the certificates I get segfaults.

julia> LibGit2.set_ssl_cert_locations("/etc/ssl/certs/Go_Daddy_Root_Certificate_Authority_G2.pem")
ERROR: ReadOnlyMemoryError()

julia> LibGit2.set_ssl_cert_locations("/etc/ssl/certs/Staat_der_Nederlanden_Root_CA.pem")

signal (11): Segmentation fault
while loading no file, in expression starting on line 0
unknown function (ip: 0x2ba68c5a7fff)
Allocations: 1671531 (Pool: 1670594; Big: 937); GC: 0
Segmentation fault

This is on

julia> versioninfo()
Julia Version 0.5.0-rc0+146
Commit 37e6397* (2016-08-03 00:47 UTC)
Platform Info:
  System: Linux (x86_64-suse-linux)
  CPU: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz
  WORD_SIZE: 64
  BLAS: libmkl_rt
  LAPACK: libmkl_rt
  LIBM: libimf
  LLVM: libLLVM-3.7.1 (ORCJIT, ivybridge)

[Keno]

I cannot reproduce this. Are there additional instruction required? Alternatively, can you post the contents of the two pem files?

[andreasnoack]

They were randomly chosen. Just pointing to the directory "/etc/ssl/certs/" was sufficient to segfault with

julia> LibGit2.set_ssl_cert_locations("/etc/ssl/certs")

Breakpoint 1, 0x00002aacbbbe7240 in git_libgit2_opts ()
   from /scratch1/scratchdirs/noack/julia/usr/bin/../lib/libgit2.so
(gdb) s
Single stepping until exit from function git_libgit2_opts,
which has no line number information.
0x00002aacbbbe71a4 in git_reflog_drop ()
   from /scratch1/scratchdirs/noack/julia/usr/bin/../lib/libgit2.so
(gdb)
Single stepping until exit from function git_reflog_drop,
which has no line number information.

Program received signal SIGSEGV, Segmentation fault.
0x0000000000000000 in ?? ()
(gdb)

[andreasnoack]

Removing the cache didn't help but updating Julia did.

[andreasnoack]

Wrong issue

[wildart]

it would be helpful if you rebuild libgit2 with debug info. Just change -DCMAKE_BUILD_TYPE=Release in deps/libgit2.mk:10 to -DCMAKE_BUILD_TYPE=Debug and run make -C deps distclean-libgit2 && make -C deps install-libgit2

[andreasnoack]

I'll try that.

[andreasnoack]

Is this useful?

julia> LibGit2.set_ssl_cert_locations("/etc/ssl/certs")

Program received signal SIGILL, Illegal instruction.
0x00002aacbbbec6b8 in git_reflog_entry_committer (entry=0x7fffffff4c48)
    at /scratch1/scratchdirs/noack/julia/deps/srccache/libgit2-211e117a0590583a720c53172406f34186c543b
---Type <return> to continue, or q <return> to quit---
d/src/reflog.c:177
177             return entry->committer;
(gdb) bt
#0  0x00002aacbbbec6b8 in git_reflog_entry_committer (entry=0x7fffffff4c48)
    at /scratch1/scratchdirs/noack/julia/deps/srccache/libgit2-211e117a0590583a720c53172406f34186c543b
d/src/reflog.c:177
#1  0x00002aaab0b216eb in julia_set_ssl_cert_locations_22823 (cert_loc=...) at libgit2/libgit2.jl:525
#2  0x00002aaab0b2173c in jlcall_set_ssl_cert_locations_22823 ()
   from /scratch1/scratchdirs/noack/julia/usr/lib/julia/sys-debug.so
#3  0x00002aaaaad1463d in jl_call_method_internal ()
   from /scratch1/scratchdirs/noack/julia/usr/bin/../lib/libjulia-debug.so.0.6
#4  0x00002aaaaad1d7be in jl_apply_generic ()
   from /scratch1/scratchdirs/noack/julia/usr/bin/../lib/libjulia-debug.so.0.6
#5  0x00002aaaaad3be13 in do_call ()
   from /scratch1/scratchdirs/noack/julia/usr/bin/../lib/libjulia-debug.so.0.6
#6  0x00002aaaaad3cbac in eval ()
   from /scratch1/scratchdirs/noack/julia/usr/bin/../lib/libjulia-debug.so.0.6

[Keno]

That backtrace makes no sense. Does this machine have perf counters? Can you run this in RR?

[andreasnoack]

I tried to build RR but the build fails because the linux/seccomp.h cannot be found.

[Keno]

Old kernel?

[andreasnoack]

Yes slightly (but recent enough that it ought to have it I think)

Linux nid00237 3.0.101-0.35.1_1.0502.8640-cray_ari_c #1 SMP Wed Feb 18 05:13:50 UTC 2015 x86_64 x86_64
 x86_64 GNU/Linux

[Keno]

rr needs at least 3.6 :(

[innerlee]

From https://discourse.julialang.org/t/pkg-system-broken-because-of-libgit2/35710/1

On Julia 1.3.1 and Ubuntu 18.04.4

$ julia
               _
   _       _ _(_)_     |  Documentation: https://docs.julialang.org
  (_)     | (_) (_)    |
   _ _   _| |_  __ _   |  Type "?" for help, "]?" for Pkg help.
  | | | | | | |/ _` |  |
  | | |_| | | | (_| |  |  Version 1.3.1 (2019-12-30)
 _/ |\__'_|_|_|\__'_|  |  Official https://julialang.org/ release
|__/                   |

julia> import LibGit2

julia> LibGit2.set_ssl_cert_locations("/etc/ssl/certs/ca-certificates.crt")

signal (11): Segmentation fault
in expression starting at REPL[2]:1
unknown function (ip: 0xffffffffffffffff)
Allocations: 942697 (Pool: 942478; Big: 219); GC: 1
[1]    6208 segmentation fault (core dumped)  julia

[StefanKarpinski]

Clearly segfaulting isn't good, but are you supposed to give it a file or a directory?

[innerlee]

It does not segfault when been given a directory. But a directory does not have any effect. Only the file works.

[innerlee]

To provide more info from the discourse thread, the following code works

$ julia
               _
   _       _ _(_)_     |  Documentation: https://docs.julialang.org
  (_)     | (_) (_)    |
   _ _   _| |_  __ _   |  Type "?" for help, "]?" for Pkg help.
  | | | | | | |/ _` |  |
  | | |_| | | | (_| |  |  Version 1.3.1 (2019-12-30)
 _/ |\__'_|_|_|\__'_|  |  Official https://julialang.org/ release
|__/                   |

(v1.3) pkg> up
  Updating registry at `~/.julia/registries/General`
  Updating git-repo `https://github.com/JuliaRegistries/General.git`
┌ Warning: Some registries failed to update:
│     — /home/lizz/.julia/registries/General — failed to fetch from repo
└ @ Pkg.Types /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.3/Pkg/src/Types.jl:1199
  Updating git-repo `https://github.com/JuliaPackaging/BinaryBuilder.jl.git`
ERROR: failed to fetch from https://github.com/JuliaPackaging/BinaryBuilder.jl.git, error: GitError(Code:ECERTIFICATE, Class:SSL, the SSL certificate is invalid: 0x08 - The certificate is not correctly signed by the trusted CA)

julia> import LibGit2

julia> LibGit2.set_ssl_cert_locations("/etc/ssl/certs/ca-certificates.crt")
0

julia> 

If the Pkg errors first, then we can set the location to that .crt file.

[joshjob42]

I can verify that this is the same for me on 1.4.0-rc2 and 1.3.1. I feel like this should be fixed somehow before 1.4.0 actually is released. I mean what better time to fix it? At the least it should be pushed out in a quick 1.4.1 because some (ahem, me) need to be able to set the ssl certificate location so we can use Pkg at all on our clusters/systems.

You can do a workaround with the following in your startup.jl it just makes you clone something every time you startup and it takes a long time.
using LibGit2 try LibGit2.clone("https://github.com/some_project/of_your_choosing.git", "/path/to/directory/blah") catch LibGit2.set_ssl_cert_locations("/etc/ssl/certs/ca-certificates.crt") end

[norefle]

Elementary OS 5.1.7 (based on Ubuntu 18.04) with official 1.5.0 Julia binary, the issue is still present.

[norefle]

Follow up, having the following code in startup.jl helps:

import LibGit2

LibGit2.initialize()
LibGit2.set_ssl_cert_locations("/etc/ssl/certs/ca-certificates.crt")

Therefore, there is no need to clone anything upfront, initialize should be enough.